Ex. 3.26

Forward stepwise regression enters the variable at each step that most reduces the residual sum-of-squares. LAR adjusts variables that have the most (absolute) correlation with the current residuals. Show that these two entry criteria are not necessarily the same.

[Hint: let \(\bx_{j.\mathcal{A}}\) be the \(j\)th variable, linearly adjusted for all the variables currently in the model. Show that the first criterion amounts to identifying the \(j\) for which \(\text{Cor}(\bx_{j.\mathcal{A}}, \bb{r})\) is largest in magnitude].

Soln. 3.26

The hint is derived in Ex. 3.9. Therefore, the difference between forward stepwise regression and LAR becomes clearer. The former chooses and includes the variable while the latter chooses, adjusts and then includes the same variable.

The paragraph below is cited from Section 2 in Least Angle Regression.

``The LARS procedure works roughly as follows. As with classic Forward Selection, we start with all coefficients equal to zero, and find the predictor most correlated with the response, say \(x_{j1}\) . We take the largest step possible in the direction of this predictor until some other predictor, say \(x_{j2}\) , has as much correlation with the current residual. At this point LARS parts company with Forward Selection. Instead of continuing along \(x_{j1}\) , LARS proceeds in a direction equiangular between the two predictors until a third variable \(x_{j3}\) earns its way into the “most correlated” set. LARS then proceeds equiangularly between \(x_{j1}\) , \(x_{j2}\) and \(x_{j3}\) , that is, along the “least angle direction,” until a fourth variable enters, and so on."