Ex. 3.29
Ex. 3.29
Suppose we run a ridge regression with parameter \(\lambda\) on a single variable \(X\), and get coefficient \(a\). We now include an exact copy \(X^\ast = X\), and refit our ridge regression. Show that both coefficients are identical, and derive their value. Show in general that if \(m\) copies of a variable \(X_j\) are included in a ridge regression, their coefficients are all the same.
Soln. 3.29
In this exercise, \(X\in \mathbb{R}^{N\times 1}\). The solution for one-dimensional ridge regression is
\[\begin{equation}
\hat\beta = \frac{X^Ty}{X^TX+\lambda}.\non
\end{equation}\]
When a new copy \(X^\ast=X\) is added, we need to solve
\[\begin{equation}
\min_{\beta_1, \beta_2}\|y-X^T\beta_1 - X^T\beta_2\|_2^2 + \lambda\|\beta_1\|_2^2 + \lambda\|\beta_2\|_2^2.\non
\end{equation}\]
First by symmetry, we know \(\hat\beta_1=\hat\beta_2\). Then we are essentially solving
\[\begin{equation}
\min_{\beta}\|y-2X^T\beta\|_2^2 + 2\lambda\|\beta\|_2^2,\non
\end{equation}\]
which yields
\[\begin{equation}
\hat\beta (= \hat\beta_1 = \hat\beta_2) = \frac{X^Ty}{2X^TX+\lambda}.\non
\end{equation}\]
Similar arguments lead to solution for the general case \(m\ge 2\):
\[\begin{equation}
\hat\beta_k = \frac{X^Ty}{mX^TX+\lambda}, \ k=1,...,m.\non
\end{equation}\]