4.2. Characterization of minimizers

For now we only have an abstract condition, which shows that there exist a minimizer without any explanation on how to characterize or compute one. In finite dimensions one would simply take the derivative of a convex function and set it to zero to characterize the extremal values, the minimizers. We will approach our problem in a similar manner and employ the Euler-Lagrange formalism to transform our variational minimization problem into a partial differential equation (PDE). We will formally derivate \(E\) and set the derivative to zero.

Definition 4.1 (Fréchet Derivative)

Let \(U\subset X\) be an open subset of the Banach space \(X\). A functional \(E:U\to \R\) is called Fréchet differentiable at \(u\in U\) if there exist a bounded linear operator \(A: X\to \R\) such that

\[\lim_{\|v\| \to 0}\frac{|E(u+v) - E(u) - Av|}{\|v\|}=0\,.\]

If there exist such an operator \(A\) it is unique and we say that \(\delta E(u) = A\) and call it the Fréchet derivative of \(E\) at \(u\).

We interpret this definition in the sense that the Fréchet derivative is the first order expansion of \(E\) at \(u\)

\[E(u+v) = E(u) + A(v) + \mathit{o}(\|v\|)\,.\]

Theorem 4.2 (Euler-Lagrange equation)

Let \(E\) be Fréchet differentiable, then

\[\bar{u} = \operatorname{arg\, min}_{u\in X}\,E(u) \Longrightarrow \delta E(\bar{u}) = 0\,.\]

Proof. Let \(\bar{u}\) be a minimizer of \(E\) and let \(\epsilon >0\), then it holds for any \(v\in X\) that \(E(\bar{u})\leq E(\bar{u}+\epsilon v)\). We define \(\Phi:\R\to\R\) with \(\Phi(\epsilon) = E(\bar{u}+\epsilon v)\). Since \(\bar{u}\) is a minimizer of \(E\) we have that a minimum of \(\Phi\) is given at \(\epsilon = 0\). We calculate now

\[\Phi'(0) = \frac{dE(\bar{u}+\epsilon v)}{d\epsilon}|_{\epsilon=0} = 0\,.\]

Where the total derivative of \(E\) with respect to \(\epsilon\) is given as

\[\frac{dE(\bar{u}+\epsilon v)}{d\epsilon} = \delta E(\bar{u} + \epsilon v) \frac{d(\bar{u} + \epsilon v)}{d\epsilon} = \delta E(\bar{u} + \epsilon v)v\]

Evaluating this term at \(\epsilon = 0\) yields \(\delta E(\bar{u})v = 0\) for all \(v\in X\). Thus \(\delta E(\bar{u}) = 0\). ◻

When \(E\) is convex then we know that the only critical points are global minimizers, so the reverse implication holds as well. The Euler-Lagrange equation of our minimization problem is sometimes called optimality condition. We will calculate the optimality condition for some of our examples.

Example 4.4 (Tikhonov Regularization)

Let \(E(u) = \frac{1}{2}\|u-f\|_2^2 + \frac{\alpha}{2}\|u\|_2^2\), then its derivative is given as \(\delta E(u) = u-f + \alpha u\), thus we have the optimality condition

\[u-f+\alpha u = 0\Leftrightarrow u = \frac{f}{1+\alpha}\,.\]

Incorporating a (weakly) continuous linear forward operator \(A\) in the data term, we need to define the adjoint of it first.

Definition 4.2 (Adjoint operator)

Let \(A: X \to Y\) be a linear map between two Banach spaces \(X\) and \(Y\). Then, the adjoint operator \(A^*\) of \(A\) is uniquely given by

\[\left< Ax, y \right> = \left< x, A^*y \right> \qquad \forall x\in X, y \in Y.\]

Remark 4.1 (Adjoint operator)

1. The adjoint operator can be generalized to unbounded operators between Banach spaces.

2. In finite dimensions \(\R^n\) and \(\R^m\), we have for a matrix \(A \in \R^{m\times n}\) that the adjoint is given by the transposed \(A^* = A^T\).

3. The adjoint of the gradient operator is the negative divergence following the divergence theorem \(\nabla^* = -\operatorname{div}\).

Example 4.5 (Tikhonov Regularization)

Let \(E(u) = \frac{1}{2}\|Au-f\|_2^2 + \frac{\alpha}{2}\|u\|_2^2\), then its derivative is given as \(\delta E(u) = A^*(Au-f) + \alpha u\), thus we have the optimality condition

\[A^*(Au-f)+\alpha u = 0\Leftrightarrow u = (A^*A + \alpha \mathbf{1})^{-1}A^*f\,.\]

Example 4.6 (Gradient Regularization)

Let \(E(u) = \frac{1}{2}\|Au-f\|_2^2 + \frac{\alpha}{2}\|\nabla u\|_2^2\), then its derivative is given as \(\delta E(u) = A^*(Au-f) + \alpha (-\operatorname{div}(\nabla u)) = A^*(Au-f)-\alpha \Delta u\), thus we have the optimality condition

\[A^*(Au-f)-\alpha\Delta u = 0\Leftrightarrow u = (A^*A - \alpha \Delta)^{-1}A^*f\,.\]

We see that the optimality condition of the gradient regularization involves solving a linear PDE. Solving the optimality condition (in form of a PDE) directly yields the startegy first optimize, then discretize which requierd calculus of variation and yields results on existence and regularity of the solution. Another approach is discretize first, then optimize which depends on numerics of finite differences and solving a (linear) system of equations.