Ex. 18.7

Consider a linear regression problem where \(p\gg N\), and assume the rank of \(\bX\) is \(N\). Let the SVD of \(\bX=\bb{U}\bb{D}\bb{V}^T = \bb{R}\bb{V}^T\),where \(\bb{R}\) is \(N\times N\) nonsingular, and \(\bb{V}\) is \(p\times N\) with orthonormal columns.

(a) Show that there are infinitely many least-squares solutions all with zero residuals.

(b) Show that the ridge-regression estimate for \(\beta\) can be written

\[\begin{equation} \hat\beta_\lambda = \bb{V}(\bb{R}^T\bb{R}+\lambda \bb{I})^{-1}\bb{R}^T\by\non \end{equation}\]

(c) Show that when \(\lambda=0\), the solution \(\hat\beta_0 = \bb{V}\bb{D}^{-1}\bb{U}^T\by\) has residuals all equal to zero, and is unique in that it has the smallest Euclidean norm amongst all zero-residual solutions.

Soln. 18.7

(a) Since \(\bX\in\mathbb{R}^{p\times N}\) has rank \(N\le p\), we know there exists \(\alpha\neq 0\) such that \(\bX\alpha = 0\). If \(\hat\beta_0\) has zero residuals, so does \(\hat\beta_0 + k\alpha\) for any \(k\in\mathbb{R}\). Therefore there are infinitely many least-squares solutions all with zero residuals.

(b) This is the same as Ex. 18.4.

(c) Note that

\[\begin{equation} \bX\hat\beta_0 = \bU\bD\bV^T\bV\bD^{-1}\bU^T\by = \by,\non \end{equation}\]

so \(\hat\beta_0\) has zero residual.

Suppose that \(\hat\beta_0+\beta\) has zero residual for some \(\beta \neq \bb{0}\), that is, \(\bX(\hat\beta_0+\beta) = \by\). Since \(\hat\beta_0\) has zero residual, we know

\[\begin{equation} \bX\beta = \bR\bV^T\beta=0.\non \end{equation}\]

Note that \(\bR\) is \(N\times N\) nonsingular, so we have \(\bV^T\beta = 0\). Now consider the Euclidean norm of \(\hat\beta_0+\beta\), we have

\[\begin{eqnarray} &&(\hat\beta_0+\beta)^T(\hat\beta_0+\beta)\non\\ &=&\hat\beta_0^T\hat\beta_0 + \beta^T\beta + 2 \hat\beta_0^T\beta\non\\ &=&\hat\beta_0^T\hat\beta_0 + \beta^T\beta + 2\by^T\bU\bD^{-1}\bV^T\beta\non\\ &=&\hat\beta_0^T\hat\beta_0 + \beta^T\beta+0.\non \end{eqnarray}\]

Since \(\beta^T\beta > 0\), we know that \(\hat\beta_0\) has the smallest Euclidean norm.