Ex. 3.19

Ex. 3.19

Show that $\Vert {\hat{\beta }}^{\text{ridge}}\Vert$ increases as its tuning parameter $\lambda \to 0$. Does the same property hold for the lasso and partial least squares estimates? For the latter, consider the tuning parameter to be the successive steps in the algorithm.

Soln. 3.19

Recall the SVD decomposition of $\mathbf{\text{X}}=\mathbf{\text{U}}\mathbf{\text{D}}{\mathbf{\text{V}}}^T$. Here $\mathbf{\text{U}}$ and $\mathbf{\text{V}}$ are $N\times p$ and $p\times p$ orthogonal matrices, and $\mathbf{\text{D}}$ is a $p\times p$ diagonal matrix. So we have

$$\begin{array}{rcl}{\beta }^{\text{ridge}}& =& ({\mathbf{\text{X}}}^T\mathbf{\text{X}}+\lambda \mathbf{\text{I}}{)}^{-1}{\mathbf{\text{X}}}^T\mathbf{\text{y}}\\ & =& {(\mathbf{\text{V}}{\mathbf{\text{D}}}^2{\mathbf{\text{V}}}^T+\lambda \mathbf{\text{I}})}^{-1}\mathbf{\text{V}}\mathbf{\text{D}}{\mathbf{\text{U}}}^T\mathbf{\text{y}}\\ & =& (\mathbf{\text{V}}({\mathbf{\text{D}}}^2+\lambda \mathbf{\text{I}}){\mathbf{\text{V}}}^T{)}^{-1}\mathbf{\text{V}}\mathbf{\text{D}}{\mathbf{\text{U}}}^T\mathbf{\text{y}}\\ & =& {\mathbf{\text{V}}}^T({\mathbf{\text{D}}}^2+\lambda \mathbf{\text{I}}{)}^{-1}\mathbf{\text{D}}{\mathbf{\text{U}}}^T\mathbf{\text{y}}.\end{array}$$

Therefore,

$$\begin{array}{rcl}\Vert {\beta }^{\text{ridge}}{\Vert }_2^2& =& {\mathbf{\text{y}}}^T\mathbf{\text{U}}\mathbf{\text{D}}({\mathbf{\text{D}}}^2+\lambda \mathbf{\text{I}}{)}^{-1}({\mathbf{\text{D}}}^2+\lambda \mathbf{\text{I}}{)}^{-1}\mathbf{\text{D}}{\mathbf{\text{U}}}^T\mathbf{\text{y}}\\ & =& ({\mathbf{\text{U}}}^T\mathbf{\text{y}}{)}^T[\mathbf{\text{D}}({\mathbf{\text{D}}}^2+\lambda \mathbf{\text{I}}{)}^{-2}\mathbf{\text{D}}]({\mathbf{\text{U}}}^T\mathbf{\text{y}})\\ & =& \sum _{j=1}^p\frac{d_j^2({\mathbf{\text{U}}}^T\mathbf{\text{y}}{)}_j^2}{(d_j^2+\lambda {)}^2}.\end{array}$$

where $\mathbf{\text{D}}({\mathbf{\text{D}}}^2+\lambda \mathbf{\text{I}}{)}^{-2}\mathbf{\text{D}}$ represents a diagonal matrix with elements $\frac{d_j^2}{(d_j^2+\lambda {)}^2}$. Therefore we see that $\Vert {\hat{\beta }}^{\text{ridge}}\Vert$ increases as its tuning parameter $\lambda \to 0$.

For Lasso, there is no explicit solution as Ridge, however, we can start with the orthogonal case, where the formula is given in Ex. 3.16. It's easy to see that $\Vert {\hat{\beta }}^{\text{lasso}}{\Vert }_1$ increases as $\lambda \to 0$. For the general case, recall the dual form of Lasso defined in (3.51) and (3.52) in the text. It's easy to see that $t$ in (3.51) and $\lambda$ in (3.52) have an inverse relationship, therefore, as $\lambda \to 0$, $t$ increases and so does the norm of optimal solutions (see Figure 3.11 for an intuitive illustration in ${\mathbb{R}}^2$).