Ex. 3.13

Ex. 3.13

Derive the expression (3.62), and show that ${\hat{\beta }}^{\text{pcr}}(p)={\hat{\beta }}^{\text{ls}}$.

Soln. 3.13

Since ${\mathbf{\text{z}}}_m=\mathbf{\text{X}}{v}_m$, from (3.61) in the text we have

$$\begin{array}{rcl}{\widehat{\mathbf{\text{y}}}}_{(M)}^{\text{pcr}}& =& \overline{y}\mathbf{\text{1}}+\mathbf{\text{X}}\sum _{m=1}^M{\hat{\theta }}_m{v}_m\\ & =& \left(\begin{array}{cc}\mathbf{\text{1}}& \mathbf{\text{X}}\end{array}\right)\left(\begin{array}{c}\overline{y}\\ \sum _{m=1}^M{\hat{\theta }}_m{v}_m\end{array}\right).\end{array}$$

Therefore, we can represent

$${\hat{\beta }}^{\text{pcr}}(M)=\sum _{m=1}^M{\hat{\theta }}_m{v}_m,$$

which is (3.62) in the text.

Recall the SVD decomposition of $\mathbf{\text{X}}=\mathbf{\text{U}}\mathbf{\text{D}}{\mathbf{\text{V}}}^T$. Here $\mathbf{\text{U}}$ and $\mathbf{\text{V}}$ are $N\times p$ and $p\times p$ orthogonal matrices, and $\mathbf{\text{D}}$ is a $p\times p$ diagonal matrix. We have $\mathbf{\text{X}}\mathbf{\text{V}}=\mathbf{\text{U}}\mathbf{\text{D}}$ since $\mathbf{\text{V}}$ is orthogonal, so that

$${\mathbf{\text{z}}}_m=\mathbf{\text{X}}{v}_m=d_m{u}_m,$$

where $d_m\in \mathbb{R}$ is the $m$-th diagonal element in $\mathbf{\text{D}}$ and $u_m$ is $m$-th column in $\mathbf{\text{U}}$. By definition of ${\hat{\theta }}_m=⟨{\mathbf{\text{z}}}_m,\mathbf{\text{y}}⟩/⟨{\mathbf{\text{z}}}_m,{\mathbf{\text{z}}}_m⟩$, we have ${\hat{\theta }}_m=⟨u_m,\mathbf{\text{y}}⟩/d_m$ so that

$${\hat{\beta }}^{\text{pcr}}(p)=\sum _{m=1}^M{\hat{\theta }}_m{v}_m=\mathbf{\text{V}}\left(\begin{array}{c}{\hat{\theta }}_1\\ {\hat{\theta }}_2\\ ⋮\\ {\hat{\theta }}_p\end{array}\right)=\mathbf{\text{V}}{\mathbf{\text{D}}}^{-1}{\mathbf{\text{U}}}^T\mathbf{\text{y}}.$$

On the other hand, recall $\mathbf{\text{X}}=\mathbf{\text{U}}\mathbf{\text{D}}{\mathbf{\text{V}}}^T$, by simple algebra we have

$$\begin{array}{rcl}{\hat{\beta }}^{ls}& =& ({\mathbf{\text{X}}}^T\mathbf{\text{X}}{)}^{-1}{\mathbf{\text{X}}}^T\mathbf{\text{y}}\\ & =& \mathbf{\text{V}}{\mathbf{\text{D}}}^{-1}{\mathbf{\text{U}}}^T\mathbf{\text{y}}.\end{array}$$

Therefore we have

$${\hat{\beta }}^{\text{pcr}}(p)={\hat{\beta }}^{ls}.$$