Ex. 12.4

Ex. 12.4

Suppose you perform a reduced-subspace linear discriminant analysis for a $K$-group problem. You compute the canonical variables of dimension $L\le K-1$ given by $z=U^{T}x$, where $U$ is the $p\times L$ matrix of discriminat coefficients, and $p>K$ is the dimension of $x$.

(a) If $L=K-1$ show that

$$\Vert z-{\overline{z}}_k{\Vert }^2-\Vert z-{\overline{z}}_{k^{\prime }}{\Vert }^2=\Vert x-{\overline{x}}_k{\Vert }_W^2-\Vert x-{\overline{x}}_{k^{\prime }}{\Vert }_W^2,$$

where $\Vert \cdot {\Vert }_W$ denotes Mahalanobis distance with respect to the covariance $W$.

(b) If $L<K-1$, show that the same expression on the left measures the difference in Mahalanobis squared distances for the distributions projected onto the subspace spanned by $U$.

Soln. 12.4

Consider the SVD $W=\hat{U}D{\hat{U}}^T$, and write $\hat{U}=({\hat{U}}_L:{\hat{U}}_{\mathrm{\perp }})$ where ${\hat{U}}_L$ represents the first $L\le K-1$ columns and ${\hat{U}}_{\mathrm{\perp }}$ the corresponding complement. It is easy to verify that

$$\begin{array}{rcl}{W}^{-1}& =& ({\hat{U}}_L{D}^{-1/2})({\hat{U}}_L{D}^{-1/2}{)}^T+({\hat{U}}_{\mathrm{\perp }}{D}^{-1/2})({\hat{U}}_{\mathrm{\perp }}{D}^{-1/2}{)}^T\\ & :=& U_L{U}_L^T+U_{\mathrm{\perp }}{U}_{\mathrm{\perp }}^T.\end{array}$$

Therefore, we have

$$\begin{array}{rcl}\Vert x-{\overline{x}}_k{\Vert }_W^2& =& \Vert U_L^T(x-{\overline{x}}_k){\Vert }^2+\Vert U_{\mathrm{\perp }}^T(x-{\overline{x}}_k){\Vert }^2\\ & =& \Vert z-{\overline{z}}_k{\Vert }^2+\Vert U_{\mathrm{\perp }}^T(x-{\overline{x}}_k){\Vert }^2.\end{array}$$

When $L=K-1$, the second term vanishes, and we recover (a).

When $L<K-1$, the first term $\Vert z-{\overline{z}}_k{\Vert }^2$ is just the Mahalanobis squared distances for the distributions projected onto the subspace spanned by $U$.