Ex. 18.15

Ex. 18.15

Kernel PCA. In Section 18.5.2 we show how to compute the principal component variables \(\bb{Z}\) from an uncentered inner-product matrix \(\bb{K}\). We compute the eigen-decomposition \((\bI-\bb{M})\bb{K}(\bI-\bb{M})=\bU\bD^2\bU^T\), with \(\bb{M}=\bb{1}\bb{1}^T/N\), and then \(\bb{Z}=\bU\bD\). Suppose we have the inner-product vector \(\bb{k}_0\), containing the \(N\) inner-products between a new point \(x_0\) and each of the \(x_i\) in our training set. Show that the (centered) projections of \(x_0\) onto the principal-component directions are given by

\[\begin{equation} \bb{z}_0=\bD^{-1}\bU^T(\bI-\bb{M})[\bb{k}_0-\bb{K}\bb{1}/N].\non \end{equation}\]

Soln. 18.15

Note that \(\tilde{\mathbb{X}} = (\bI-\bb{M})\bX = \bb{U}\bb{D}\bb{V}^T\), we have \((\bI-\bb{M})\bX\bb{V} = \bb{U}\bb{D}\) since \(\bb{V}^T\bb{V}=\bb{I}\). Thus we can write

\[\begin{equation} \bb{z}_0 = \bb{V}^T(\bb{x}_0-\bar x)\in \mathbb{R}^{N\times 1}.\non \end{equation}\]

On the other hand, we have

\[\begin{eqnarray} \bb{U}\bb{D}\bb{z}_0 &=& \bb{U}\bb{D}\bb{V}^T(\bb{x}_0-\bar x)\non\\ &=& (\bI-\bb{M})\bX (\bb{x}_0-\bar x)\non\\ & =& (\bI-\bb{M})(\bb{k}_0-\bb{K}\bb{1}/N).\non \end{eqnarray}\]

Therefore we obtain

\[\begin{equation} \bb{z}_0=\bD^{-1}\bU^T(\bI-\bb{M})[\bb{k}_0-\bb{K}\bb{1}/N].\non \end{equation}\]