(characterization-of-minimizers)=
# Characterization of minimizers

For now we only have an abstract condition, which shows that there exist
a minimizer without any explanation on how to characterize or compute
one. In finite dimensions one would simply take the derivative of a
convex function and set it to zero to characterize the extremal values,
the minimizers. We will approach our problem in a similar manner and
employ the *Euler-Lagrange* formalism to transform our variational
minimization problem into a *partial differential equation* (PDE). We
will formally derivate $E$ and set the derivative to zero.

````{prf:definition} Fréchet Derivative
Let $U\subset X$ be an open subset of the Banach space $X$. A functional
$E:U\to \R$ is called Fréchet differentiable at $u\in U$ if there exist
a bounded linear operator $A: X\to \R$ such that
```{math}
\lim_{\|v\| \to 0}\frac{|E(u+v) - E(u) - Av|}{\|v\|}=0\,.
```
If there exist such an operator $A$ it is unique and we say that
$\delta E(u) = A$ and call it the Fréchet derivative of $E$ at $u$.

````

We interpret this definition in the sense that the Fréchet derivative is
the first order expansion of $E$ at $u$
```{math}
E(u+v) = E(u) + A(v) + \mathit{o}(\|v\|)\,.
```

````{prf:theorem} Euler-Lagrange equation
Let $E$ be Fréchet differentiable, then
```{math}
\bar{u} = \operatorname{arg\, min}_{u\in X}\,E(u) \Longrightarrow \delta E(\bar{u}) = 0\,.
```

````

````{prf:proof} 
Let $\bar{u}$ be a minimizer of $E$ and let $\epsilon >0$, then it holds
for any $v\in X$ that $E(\bar{u})\leq E(\bar{u}+\epsilon v)$. We define
$\Phi:\R\to\R$ with $\Phi(\epsilon) = E(\bar{u}+\epsilon v)$. Since
$\bar{u}$ is a minimizer of $E$ we have that a minimum of $\Phi$ is
given at $\epsilon = 0$. We calculate now
```{math}
\Phi'(0) = \frac{dE(\bar{u}+\epsilon v)}{d\epsilon}|_{\epsilon=0} = 0\,.
```
Where the total derivative of $E$ with respect to $\epsilon$ is given as
```{math}
\frac{dE(\bar{u}+\epsilon v)}{d\epsilon} = \delta E(\bar{u} + \epsilon v) \frac{d(\bar{u} + \epsilon v)}{d\epsilon} = \delta E(\bar{u} + \epsilon v)v
```
Evaluating this term at $\epsilon = 0$ yields $\delta E(\bar{u})v = 0$
for all $v\in X$. Thus $\delta E(\bar{u}) = 0$. ◻

````

When $E$ is convex then we know that the only critical points are global
minimizers, so the reverse implication holds as well. The Euler-Lagrange
equation of our minimization problem is sometimes called optimality
condition. We will calculate the optimality condition for some of our
examples.

````{prf:example} Tikhonov Regularization
Let $E(u) = \frac{1}{2}\|u-f\|_2^2 + \frac{\alpha}{2}\|u\|_2^2$, then
its derivative is given as $\delta E(u) = u-f + \alpha u$, thus we have
the optimality condition
```{math}
u-f+\alpha u = 0\Leftrightarrow u = \frac{f}{1+\alpha}\,.
```

````

Incorporating a (weakly) continuous linear forward operator $A$ in the
data term, we need to define the adjoint of it first.

````{prf:definition} Adjoint operator
Let $A: X \to Y$ be a linear map between two Banach spaces $X$ and $Y$.
Then, the adjoint operator $A^*$ of $A$ is uniquely given by
```{math}
\left< Ax, y \right> = \left< x, A^*y \right> \qquad \forall x\in X, y \in Y.
```

````

````{prf:remark} Adjoint operator

1.  The adjoint operator can be generalized to unbounded operators
    between Banach spaces.

2.  In finite dimensions $\R^n$ and $\R^m$, we have for a matrix
    $A \in \R^{m\times n}$ that the adjoint is given by the transposed
    $A^* = A^T$.

3.  The adjoint of the gradient operator is the negative divergence
    following the divergence theorem $\nabla^* = -\operatorname{div}$.

````

````{prf:example} Tikhonov Regularization
Let $E(u) = \frac{1}{2}\|Au-f\|_2^2 + \frac{\alpha}{2}\|u\|_2^2$, then
its derivative is given as $\delta E(u) = A^*(Au-f) + \alpha u$, thus we
have the optimality condition
```{math}
A^*(Au-f)+\alpha u = 0\Leftrightarrow u = (A^*A + \alpha \mathbf{1})^{-1}A^*f\,.
```

````

````{prf:example} Gradient Regularization
Let $E(u) = \frac{1}{2}\|Au-f\|_2^2 + \frac{\alpha}{2}\|\nabla u\|_2^2$,
then its derivative is given as
$\delta E(u) = A^*(Au-f) + \alpha (-\operatorname{div}(\nabla u)) = A^*(Au-f)-\alpha \Delta u$,
thus we have the optimality condition
```{math}
A^*(Au-f)-\alpha\Delta u = 0\Leftrightarrow u = (A^*A - \alpha \Delta)^{-1}A^*f\,.
```

````

We see that the optimality condition of the gradient regularization
involves solving a linear PDE. Solving the optimality condition (in form
of a PDE) directly yields the startegy **first optimize, then
discretize** which requierd calculus of variation and yields results on
existence and regularity of the solution. Another approach is
**discretize first, then optimize** which depends on numerics of finite
differences and solving a (linear) system of equations.