Ex. 12.4

Suppose you perform a reduced-subspace linear discriminant analysis for a \(K\)-group problem. You compute the canonical variables of dimension \(L\le K-1\) given by \(z = U^Tx\), where \(U\) is the \(p\times L\) matrix of discriminat coefficients, and \(p > K\) is the dimension of \(x\).

(a) If \(L=K-1\) show that

\[\begin{equation} \|z-\bar z_k\|^2 - \|z-\bar z_{k'}\|^2 = \|x-\bar x_k\|^2_W - \|x-\bar x_{k'}\|^2_W,\non \end{equation}\]

where \(\|\cdot\|_W\) denotes Mahalanobis distance with respect to the covariance \(W\).

(b) If \(L < K-1\), show that the same expression on the left measures the difference in Mahalanobis squared distances for the distributions projected onto the subspace spanned by \(U\).

Soln. 12.4

Consider the SVD \(W=\hat U D \hat U^T\), and write \(\hat U = (\hat U_L : \hat U_\bot)\) where \(\hat U_L\) represents the first \(L\le K-1\) columns and \(\hat U_\bot\) the corresponding complement. It is easy to verify that

\[\begin{eqnarray} W^{-1} &=& (\hat U_LD^{-1/2})(\hat U_LD^{-1/2})^T + (\hat U_\bot D^{-1/2})(\hat U_\bot D^{-1/2})^T\non\\ &:=& U_LU_L^T + U_\bot U_\bot ^T.\non \end{eqnarray}\]

Therefore, we have

\[\begin{eqnarray} \|x-\bar x_k\|_W^2 &=& \|U_L^T(x-\bar x_k)\|^2 + \|U_\bot^T(x-\bar x_k)\|^2\non\\ &=&\|z-\bar z_k\|^2+ \|U_\bot^T(x-\bar x_k)\|^2.\non \end{eqnarray}\]

When \(L=K-1\), the second term vanishes, and we recover (a).

When \(L<K-1\), the first term \(\|z-\bar z_k\|^2\) is just the Mahalanobis squared distances for the distributions projected onto the subspace spanned by \(U\).