Ex. 7.4

Ex. 7.4

Consider the in-sample prediction error (7.18) and the training error $\overline{\text{err}}$ in the case of squared-error loss:

$$\begin{array}{rcl}{\text{Err}}_{\text{in}}& =& \frac{1}{N}\sum _{i=1}^N{E}_{Y^0}(Y_i^0-\hat{f}(x_i){)}^2\\ \overline{\text{err}}& =& \frac{1}{N}\sum _{i=1}^N(y_i-\hat{f}(x_i){)}^2.\end{array}$$

Add and subtract $f(x_i)$ and $E\hat{f}(x_i)$ in each expression and expand. Hence establish that the average optimism in the training error is

$$\frac{2}{N}\sum _{i=1}^N\text{Cov}({\hat{y}}_i,y_i),$$

as given in (7.21).

Soln. 7.4

We start with ${\text{Err}}_{\text{in}}$. Let's denote ${\hat{y}}_i=\hat{f}(x_i)$ and write

$$Y_i^0-\hat{f}(x_i)=Y_i^0-f(x_i)+f(x_i)-E{\hat{y}}_i+E{\hat{y}}_i-{\hat{y}}_i$$

so that

$$\begin{array}{rcl}{\text{Err}}_{\text{in}}& =& \frac{1}{N}\sum _{i=1}^N{E}_{Y^0}{(Y_i^0-f(x_i)+f(x_i)-E{\hat{y}}_i+E{\hat{y}}_i-{\hat{y}}_i)}^2\\ & =& \frac{1}{N}\sum _{i=1}^N{A}_i+B_i+C_i+D_i+E_i+F_i,\end{array}$$

where

$$\begin{array}{rcl}{A}_i& =& E_{Y^0}(Y_i^0-f(x_i){)}^2\\ B_i& =& E_{Y^0}(f(x_i)-E{\hat{y}}_i{)}^2=(f(x_i)-E{\hat{y}}_i{)}^2\\ C_i& =& E_{Y^0}(E{\hat{y}}_i-{\hat{y}}_i{)}^2=(E{\hat{y}}_i-{\hat{y}}_i{)}^2\\ D_i& =& 2E_{Y^0}(Y_i^0-f(x_i))(f(x_i)-E{\hat{y}}_i)\\ E_i& =& 2E_{Y^0}(Y_i^0-f(x_i))(E{\hat{y}}_i-{\hat{y}}_i)\\ F_i& =& 2E_{Y^0}(f(x_i)-E{\hat{y}}_i)(E{\hat{y}}_i-{\hat{y}}_i)=2(f(x_i)-E{\hat{y}}_i)(E{\hat{y}}_i-{\hat{y}}_i)\end{array}$$

Similarly for $\overline{\text{err}}$ we have

$$y_i-\hat{f}(x_i)=y_i-f(x_i)+f(x_i)-E{\hat{y}}_i+E{\hat{y}}_i-{\hat{y}}_i$$

and

$$\begin{array}{rcl}\overline{\text{err}}& =& \frac{1}{N}\sum _{i=1}^N(y_i-f(x_i)+f(x_i)-E{\hat{y}}_i+E{\hat{y}}_i-{\hat{y}}_i{)}^2\\ & =& \frac{1}{N}\sum _{i=1}^N{G}_i+B_i+C_i+H_i+J_i+F_i,\end{array}$$

where

$$\begin{array}{rcl}{G}_i& =& (y_i-f(x_i){)}^2\\ H_i& =& 2(y_i-f(x_i))(f(x_i)-E{\hat{y}}_i)\\ J_i& =& 2(y_i-f(x_i))(E{\hat{y}}_i-{\hat{y}}_i).\end{array}$$

Therefore, we have

$$\begin{array}{rcl}{E}_{\mathbf{\text{y}}}(\text{op})& =& E_{\mathbf{\text{y}}}({\text{Err}}_{\text{in}}-\overline{\text{err}})\\ & =& \frac{1}{N}\sum _{i=1}^N{E}_{\mathbf{\text{y}}}[(A_i-G_i)+(D_i-H_i)+(E_i-J_i)].\end{array}$$

For $A_i$ and $G_i$, the expectaion over $\mathbf{\text{y}}$ captures unpredictable error and thus $E_{\mathbf{\text{y}}}(A_i-G_i)=0$. Similarly we have $E_{\mathbf{\text{y}}}{D}_i=E_{\mathbf{\text{y}}}{H}_i=E_{\mathbf{\text{y}}}{E}_i=0$, and thus

$$\begin{array}{rcl}{E}_{\mathbf{\text{y}}}(\text{op})& =& -\frac{2}{N}\sum _{i=1}^N{J}_i\\ & =& -\frac{2}{N}\sum _{i=1}^N{E}_{\mathbf{\text{y}}}(y_i-f(x_i))(E{\hat{y}}_i-{\hat{y}}_i)\\ & =& \frac{2}{N}\sum _{i=1}^N[E_{\mathbf{\text{y}}}(y_i{\hat{y}}_i)-E_{\mathbf{\text{y}}}{y}_i{E}_{\mathbf{\text{y}}}{\hat{y}}_i]\\ & =& 2\text{Cov}(y_i,{\hat{y}}_i).\end{array}$$