Ex. 14.20

Fixed point algorithm for ICA (Hyv\(\ddot{a}\)rinen et al., 2001) Consider maximizing \(C(a) = E\{g(a^TX)\}\) with respect to \(a\), with \(\|a\|=1\) and \(\text{Cov}(X)=I\). Use a Lagrange multiplier to enforce the norm constraint, and write down the first two derivatives of the modified criterion. Use the approximation

\[\begin{equation} E\{XX^Tg^{''}(a^TX)\}\approx E\{XX^T\}E\{g^{''}(a^TX)\}\non \end{equation}\]

to show that the Newton update can be written as the fixed-point update (14.96).

Soln. 14.20

The Lagrangian multiplier is

\[\begin{equation} L(a, \lambda) = E[g(a^T\bX)] + \lambda(a^Ta-1).\non \end{equation}\]

We have

\[\begin{eqnarray} \frac{\partial L(a, \lambda)}{\partial a} &=& E[g'(a^T\bX)\bX] + 2 \lambda a\non \end{eqnarray}\]

and

\[\begin{eqnarray} \frac{\partial^2 L(a, \lambda)}{\partial a \partial a^T} &=& E[g''(a^T\bX)\bX\bX^T] + 2\lambda.\non \end{eqnarray}\]

Newton's method performs the iteration

\[\begin{equation} x_{k+1} = x_k - \frac{f'(x_k)}{f''(x_k)}.\non \end{equation}\]

Therefore, in this case, we know

\[\begin{eqnarray} a_j &\leftarrow& a_j - \frac{E[g'(a_j^T\bX)\bX] + 2 \lambda a_j}{E[g''(a_j^T\bX)\bX\bX^T] + 2\lambda}\non\\ &\approx& a_j - \frac{E[g'(a_j^T\bX)\bX] + 2 \lambda a_j}{E[g''(a_j^T\bX)]E[\bX\bX^T] + 2\lambda}\non\\ &=&\frac{a_jE[g''(a_j^T\bX)] + 2\lambda a_j - E[g'(a_j^T\bX)\bX] - 2 \lambda a_j}{E[g''(a_j^T\bX)] + 2\lambda} \ \text{ ( note } \text{Cov}(\bX) = \bb{I}\ )\non\\ &\propto&E\{\bX g'(a_j^T\bX) - E[g''(a^T_j\bX)]a_j\}.\non \end{eqnarray}\]

Therefore the Newton update can be written as the fixed-point update (14.96) in the text (followed by the orthogonalization of \(\bA\)).