Ex. 8.1

Let \(r(y)\) and \(q(y)\) be the probability density functions. Jensen's inequality states that for a random variable \(X\) and a convex function \(\phi(x)\), \(E[\phi(x)]\ge \phi(E[X])\). Use Jensen's inequality to show that

\[\begin{equation} E_q\log[r(Y)/q(Y)]\non \end{equation}\]

is maximized as a function of \(r(y)\) when \(r(y) = q(y)\). Hence show that \(R(\theta, \theta)\ge R(\theta', \theta)\) as stated below equation (8.46).

Soln. 8.1

Note that \(-\log(x)\) is convex, by Jensen's inequality, we have

\[\begin{eqnarray} E_q[-\log[r(Y)/q(Y)]] &\ge& -\log[E_q[r(Y)/q(Y)]]\non\\ &=&-\log\left[\int\frac{r(y)}{q(y)}q(y)dy\right]\non\\ &=&-\log\left[\int r(y)dy\right]\non\\ &=&-\log(1)\non\\ &=&0,\non \end{eqnarray}\]

therefore we have

\[\begin{equation} E_q[\log(r(Y)/q(Y))]\le 0 = E_q[\log(q(Y)/q(Y))].\non \end{equation}\]

So the expectation is maximized when \(r = q\).

For equation (8.46) in the text, we have

\[\begin{eqnarray} R(\theta', \theta) - R(\theta, \theta) &=& E[\ell_1(\theta;\bb{Z}^m|\bb{Z})|\bb{Z}, \theta] - E[\ell(\theta;\bb{Z}^m|\bb{Z})|\bb{Z}, \theta]\non\\ &=&E_{\text{Pr}(\bb{Z}^m|\bb{Z}, \theta)}\left(\log \frac{\text{Pr}(\bb{Z}^m|\bb{Z}, \theta')}{\text{Pr}(\bb{Z}^m|\bb{Z}, \theta)}\right)\non\\ &\le&0.\non \end{eqnarray}\]