3.1. Maximum likelihood#
We have determined the conditional probability \(p(f\mid u)\) for
different noise models, we are now interested in the maximum
likelihood.
Idea: Use maximum likelihood estimator (MLE) to estimate the most
likely \(u\) under observation of \(f\)
We will use the notion of the negative log-likelihood to transfer the maximization problem to a minimization problem
This can be done due to the monotonicity of the log function. It is a useful convention due to the exponential structure of the conditional probabilities of our noise models.
Additive Gaussian noise:
We have the following probability \(p(f_i\mid u_i)=\frac{1}{\sqrt{2 \pi \sigma^2}} \cdot e^{\frac{-|f_i-ui|^2}{2 \sigma^2}}\) and calculate its negative log\[-\log{(p(f_i\mid u_i))} = -\log{\left(\frac{1}{\sqrt{2 \pi \sigma^2}}\right)} - \left(-\frac{|f_i-u_i|^2}{2 \sigma^2}\right) = -\log{\left(\frac{1}{\sqrt{2 \pi \sigma^2}}\right)} + \frac{|f_i -u_i|^2}{2 \sigma^2}\,.\]Using now the maximum likelihood estimator and its negative log, yields with our assumptions
\[\begin{aligned} &\operatorname{arg\,max}_{u \in \R^N}{p(F=f_i|U=u_i)}=\operatorname{arg\,max}_{u \in \R^N} \prod_{i=1}^{N} \frac{p(f_i,u_i)}{p(u_i)} \\ \Leftrightarrow &\operatorname{arg\,min}_{u\in \R^N}{-\log{p(F=f_i|U=u_i)}}= \operatorname{arg\,min}_{u\in \R^N}-\sum_{i=1}^N\log\left(\frac{p(f_i\mid u_i)}{p(u_i)} \right)\\ &= \operatorname{arg\,min}_{u\in \R^N}-\sum_{i=1}^N \left( \log(p(f_i\mid u_i)) - \log(p(u_i)) \right)\\ &= \operatorname{arg\,min}_{u\in \R^N}\sum_{i=1}^N -\log{\left(\frac{1}{\sqrt{2 \pi \sigma^2}}\right)} + \frac{|f_i -u_i|^2}{2 \sigma^2} + \log(p(u_i)) \\ &=\operatorname{arg\,min}_{u\in \R^N}\sum_{i=1}^N -\log{\left(\frac{1}{\sqrt{2 \pi \sigma^2}}\right)} + \frac{|f_i -u_i|^2}{2 \sigma^2} + \log(\frac{1}{N}) \\ &= \operatorname{arg\,min}_{u\in \R^N}\sum_{i=1}^N\frac{|f_i -u_i|^2}{2 \sigma^2}\\ &= \operatorname{arg\,min}_{u\in \R^N} \frac{\|f-u\|_2^2}{2\sigma^2} = f \end{aligned}\]Poisson noise:
We calculate the negative log of \(p(f_i\mid u_i)= \frac{u_i^{f_i}\cdot e^{-u_i}}{f_i!}\)\[-\log{p\left(f_i,u_i\right)}=-f_i\cdot \log{u_i}+u_i+\log{(f_i!)}\]and estimate the maximum likelihood
\[\begin{aligned} &\operatorname{arg\,min}_{u\in \R^N}-\sum_{i=1}^N \left( \log(p(f_i\mid u_i)) - \log(p(u_i)) \right)\\ = &\operatorname{arg\,min}_{u\in \R^N}\sum_{i=1}^N \left( -f_i\cdot \log{u_i}+u_i+\log{(f_i!)} + \log(\frac{1}{N}) \right)\\ = &\operatorname{arg\,min}_{u\in \R^N}\sum_{i=1}^N \left( -f_i\cdot \log{u_i}+u_i\right)\\ = &\operatorname{arg\,min}_{u\in \R^N}\langle \mathbf{1}, u\rangle - \langle f, \log(u)\rangle = f \end{aligned}\]