Ex. 3.30

Consider the elastic-net optimization problem:

\[\begin{equation} \min_\beta \|\by-\bX\beta\|^2 + \lambda\left[\alpha \|\beta\|_2^2 + (1-\alpha)\|\beta\|_1\right].\non \end{equation}\]

Show how one can turn this into a lasso problem, using an augmented version of \(\bX\) and \(\by\).

Soln. 3.30

Assume \(\by\in\mathbb{R}^{N\times 1}\), \(\bX\in\mathbb{R}^{N\times (p+1)}\) and \(\beta\in\mathbb{R}^{(p+1)\times 1}\). We first augment \(\bX\) by

\[\begin{equation} \tilde\bX = \begin{pmatrix} \bX\\ \gamma \bb{I}_{p+1} \end{pmatrix}\in \mathbb{R}^{(N+p+1)\times (p+1)}\non \end{equation}\]

for \(\gamma > 0\). Then we augment \(\by\) by

\[\begin{equation} \tilde \by = \begin{pmatrix} \by\\ \bb{0}_{p+1} \end{pmatrix}\in\mathbb{R}^{(N+p+1)\times 1}.\non \end{equation}\]

Then we have

\[\begin{equation} \|\tilde \by - \tilde\bX\beta\|_2^2 = \left\|\begin{pmatrix} \by-\bX\beta\\ \gamma\beta \end{pmatrix}\right\|_2^2 = \|\by-\bX\beta\|_2^2 + \gamma^2\|\beta\|_2^2.\non \end{equation}\]

So consider the lasso problem for \((\tilde\by, \tilde\bX)\)

\[\begin{equation} \min_{\beta} \|\tilde \by - \tilde\bX\beta\|_2^2 + \delta\|\beta\|_1,\non \end{equation}\]

which is essentially

\[\begin{equation} \min_{\beta}\|\by-\bX\beta\|_2^2 + \gamma^2\|\beta\|_2^2 + \delta\|\beta\|_1.\non \end{equation}\]

By choosing \(\gamma = \sqrt{\lambda\alpha}\) and \(\delta = \lambda(1-\alpha)\) we get the original elastic-net problem.

Remark

This exercise is similar to Ex. 3.12.