Ex. 3.5
Ex. 3.5
Consider the ridge regression problem (3.41). Show that this problem is equivalent to the problem
\[\begin{equation}
\hat\beta^c = \underset{\beta^c}{\operatorname{arg min}}\left\{\sum_{i=1}^N[y_i-\beta_0^c - \sum_{j=1}^p(x_{ij}-\bar x_j)\beta^c_j]^2 + \lambda \sum_{j=1}^p(\beta_j^c)^2\right\}.\nonumber
\end{equation}\]
Given the correspondence between \(\beta^c\) and the original \(\beta\) in (3.41). Characterize the solution to this modified criterion. Show that a similar result holds for the lasso.
Soln. 3.5
We center each \(x_{ij}\) by replacing \(x_{ij}\) with \(x_{ij}-\bar x_j\), then (3.41) becomes
\[\begin{equation}
\hat\beta^{\text{ridge}} = \underset{\beta}{\operatorname{arg min}}\left\{\sum_{i=1}^N[y_i - \beta_0 - \sum_{j=1}^p\bar x_j\beta_j - \sum_{j=1}^p(x_{ij}-\bar x_j)\beta_j]^2 + \lambda \sum_{j=1}^p\beta_j^2\right\}.\nonumber
\end{equation}\]
Looking at the two problems, we can see \(\beta^c\) can be transformed from original \(\beta\) as
\[\begin{eqnarray}
\beta_0^c &=& \beta_0 + \sum_{j=1}^p\bar x_j\beta_j\nonumber\\
\beta_j^c &=& \beta_j \ \text{ for } j =1,...,p.\nonumber
\end{eqnarray}\]
It's easy to see that exact same centering technique applies to the lasso.
To characterize the solution, we first take derivative w.r.t \(\beta_0^c\) and set it equal to 0, which yields
\[\begin{equation}
\sum_{i=1}^N\left(y_i-\beta_0^c - \sum_{j=1}^p(x_{ij}-\bar x_j)\beta_j\right) = 0,\nonumber
\end{equation}\]
which further implies that \(\beta_0^c=\bar y\). Next we set
\[\begin{eqnarray}
\tilde y_i &=& y_i -\beta_0^c,\nonumber\\
\tilde x_{ij} &=& x_{ij} - \bar x_j,\nonumber
\end{eqnarray}\]
the problem, in matrix form, becomes
\[\begin{equation}
\min_{\beta^c} (\tilde{\textbf{y}} - \tilde{\textbf{X}}\beta^c)^T(\tilde{\textbf{y}} - \tilde{\textbf{X}}\beta^c) + \lambda\beta_c^T\beta_c.\nonumber
\end{equation}\]
It's easy to see the solution is
\[\begin{equation}
\hat\beta_c = (\tilde{\textbf{X}}^T\tilde{\textbf{X}}+\lambda\textbf{I})^{-1}\tilde{\textbf{X}}^T\textbf{y}.\nonumber
\end{equation}\]