Ex. 3.15

Ex. 3.15

Verify expression (3.64), and hence show that the partial least squares directions are a compromise between the ordinary regression coefficient and the principal component directions.

Soln. 3.15

Note that

$${\text{Corr}}^2(\mathbf{\text{y}},\mathbf{\text{X}}\alpha )\text{Var}(\mathbf{\text{X}}\alpha )=\frac{{\text{Cov}}^2(\mathbf{\text{y}},\mathbf{\text{X}}\alpha )}{\text{Var}(\mathbf{\text{y}})\text{Var}(\mathbf{\text{X}}\alpha )}\text{Var}(\mathbf{\text{X}}\alpha )=\frac{{\text{Cov}}^2(\mathbf{\text{y}},\mathbf{\text{X}}\alpha )}{\text{Var}(\mathbf{\text{y}})}$$

We are essentially solving

$$\begin{array}{rcl}\underset{\alpha }{max}& & ({\mathbf{\text{y}}}^T\mathbf{\text{X}}\alpha {)}^2\\ \text{s.t.}& & \Vert \alpha \Vert =1\\ & & {\alpha }^T\mathbf{\text{S}}{\hat{\phi }}_l=0,l=1,...,m-1,\end{array}$$

where $\mathbf{\text{S}}={\mathbf{\text{X}}}^T\mathbf{\text{X}}$ is the sample covariance matrix of the ${\mathbf{\text{x}}}_j$.

We start with the case $m=1$, which immediately gives what we call the first canonical covariance variable (see Ex. 3.20) with

$${\hat{\alpha }}_1={\mathbf{\text{X}}}^T\mathbf{\text{y}}/\Vert {\mathbf{\text{X}}}^T\mathbf{\text{y}}{\Vert }_2.$$

Note that ${\hat{\alpha }}_1\propto {\hat{\phi }}_1$ in Algorithm 3.3 in the text.

The second canonical covariance variable, namely ${\hat{\alpha }}_2$, has to maximize the same objective with additional constraint ${\hat{\alpha }}_2^T\mathbf{\text{S}}{\hat{\alpha }}_1=0$. It turns out

$$\begin{array}{}\text{(1)}& {\hat{\alpha }}_2\propto {\mathbf{\text{X}}}^T\mathbf{\text{y}}-\left(\frac{{\mathbf{\text{y}}}^T\mathbf{\text{X}}\mathbf{\text{S}}{\mathbf{\text{X}}}^T\mathbf{\text{y}}}{{\mathbf{\text{y}}}^T\mathbf{\text{X}}{\mathbf{\text{S}}}^2{\mathbf{\text{X}}}^T\mathbf{\text{y}}}\right)\mathbf{\text{S}}{\mathbf{\text{X}}}^T\mathbf{\text{y}}.\end{array}$$

To see that, we first verify that

$$\begin{array}{rcl}{\hat{\alpha }}_2^T\mathbf{\text{S}}{\hat{\alpha }}_1& \propto & {\mathbf{\text{y}}}^T\mathbf{\text{X}}\mathbf{\text{S}}{\mathbf{\text{X}}}^T\mathbf{\text{y}}-\left(\frac{{\mathbf{\text{y}}}^T\mathbf{\text{X}}\mathbf{\text{S}}{\mathbf{\text{X}}}^T\mathbf{\text{y}}}{{\mathbf{\text{y}}}^T\mathbf{\text{X}}{\mathbf{\text{S}}}^2{\mathbf{\text{X}}}^T\mathbf{\text{y}}}\right){\mathbf{\text{y}}}^T\mathbf{\text{X}}{\mathbf{\text{S}}}^T\mathbf{\text{S}}{\mathbf{\text{X}}}^T\mathbf{\text{y}}\\ & =& 0.\end{array}$$

Second, for ${\alpha }_2$ satisfying ${\alpha }_2^T\mathbf{\text{S}}{\hat{\alpha }}_1=0$, that is, ${\alpha }_2^T\mathbf{\text{S}}{\mathbf{\text{X}}}^T\mathbf{\text{y}}=0$, we have the objective to maximize

$${\alpha }_2^T{\mathbf{\text{X}}}^T\mathbf{\text{y}}={\alpha }_2^T({\mathbf{\text{X}}}^T\mathbf{\text{y}}-\left(\frac{{\mathbf{\text{y}}}^T\mathbf{\text{X}}\mathbf{\text{S}}{\mathbf{\text{X}}}^T\mathbf{\text{y}}}{{\mathbf{\text{y}}}^T\mathbf{\text{X}}{\mathbf{\text{S}}}^2{\mathbf{\text{X}}}^T\mathbf{\text{y}}}\right)\mathbf{\text{S}}{\mathbf{\text{X}}}^T\mathbf{\text{y}}).$$

Therefore we see $\text{(1)}$ holds. Note that ${\hat{\alpha }}_2\propto {\hat{\phi }}_2$ in the Algorithm 3.3 in the text. Continuing this, we are able to derive ${\hat{\phi }}_m$ for $m\ge 1$.

Now we are ready to show that partial least squares (PLS) directions are a compromise between the ordinary regression coefficient (OLS) and the principal component directions (PCR). The regressors for OLS, PCR and PLS may be referred to as canonical correlation, canonical variance and canonical covariance variables respectively. A generalized criterion that encompasses all three methods is

$$\begin{array}{rcl}\underset{\alpha }{max}& & ({\mathbf{\text{y}}}^T\mathbf{\text{X}}\alpha {)}^2{\left({\alpha }^T{\mathbf{\text{X}}}^T\mathbf{\text{X}}\alpha \right)}^{\frac{r}{1-r}-1}\\ \text{s.t.}& & \Vert \alpha \Vert =1\\ & & {\alpha }^T\mathbf{\text{S}}{\hat{\alpha }}_l=0,l=1,...,m-1,\end{array}$$

where $r\in [0,1).$ When $r=0$ we recover OLS, and when $r\to 1$ we get PCR. The case when $r=1/2$ gives PLS. Note that this generalized regression is referred to as continuum regression. See paper Continuum Regression and Ridge Regression for more details.