Ex. 8.2
Ex. 8.2
Consider the maximization of the log-likelihood (8.48), over distributions \(\tilde P(\bb{Z}^m)\) such that \(\tilde P(\bb{Z}^m)\ge 0\) and \(\sum_{\bb{Z}^m}\tilde P(\bb{Z}^m) = 1\). Use Lagrange multipliers to show that the solution is the conditional distribution \(\tilde P(\bb{Z}^m) = \text{Pr}(\bb{Z}^m|\bb{Z}, \theta')\), as in (8.49).
Soln. 8.2
We first write
\[\begin{eqnarray}
F(\theta', \tilde P) &=& E_{\tilde P}[\ell_0(\theta';\bb{T})] - E_{\tilde P}[\log \tilde P(\bb{Z}^m)]\non\\
&=& \sum_{\bb{Z}^m} \ell_0(\theta';\bb{T})\tilde P(\bb{Z}^m)-\sum_{\bb{Z}^m}\tilde P(\bb{Z}^m)\log \tilde P(\bb{Z}^m).\non
\end{eqnarray}\]
With the constraint \(\sum_{\bb{Z}^m}\tilde P(\bb{Z}^m) = 1\), the Lagrange multiplier of \(F(\theta', \tilde P)\) with \(\theta'\) fixed is
\[\begin{equation}
L(\tilde P, \lambda) = \sum_{\bb{Z}^m} \ell_0(\theta';\bb{T})\tilde P(\bb{Z}^m)-\sum_{\bb{Z}^m}\tilde P(\bb{Z}^m)\log \tilde P(\bb{Z}^m) - \lambda\left(\sum_{\bb{Z}^m}\tilde P(\bb{Z}^m)-1\right).\non
\end{equation}\]
Further we set
\[\begin{equation}
\label{eq:ex8-2lag}
\frac{\partial L(\tilde P, \lambda)}{\partial \tilde P} = \ell_0(\theta';\bb{T}) - \left(\log \tilde P(\bb{Z}^m) + 1\right) +\lambda =0
\end{equation}\]
so that
\[\begin{equation}
\tilde P(\bb{Z}^m) =\exp(\ell_0(\theta';\bb{T}) + \lambda -1).\non
\end{equation}\]
Recall the constraint that \(\sum_{\bb{Z}^m}\tilde P(\bb{Z}^m) = 1\), we get
\[\begin{equation}
\sum_{\bb{Z}^m}\exp(\ell_0(\theta';\bb{T}) + \lambda -1) =1, \non
\end{equation}\]
which yields
\[\begin{equation}
\lambda = 1 - \log\left(\text{Pr}(\bb{Z}|\theta')\right).\non
\end{equation}\]
Plugging \(\lambda\) above into \(\eqref{eq:ex8-2lag}\) we get
\[\begin{equation}
\tilde P(\bb{Z}^m) = \text{Pr}(\bb{Z}^m|\bb{Z}, \theta').\non
\end{equation}\]