Ex. 2.5

Ex. 2.5
(a)

Derive equation (2.27). The last line makes use of (3.8) through a conditioning argument.

(b)

Derive equation (2.28), making use of the cyclic property of the trace operator [trace(\(AB\)) = trace(\(BA\))], and its linearity (which allows us to interchange the order of trace and expectation).

Soln. 2.5
(a)

Since \(X\) and \(\epsilon\) are independent, we have \(E_{\mathcal{T}}(\epsilon) = 0\). Since \(\hat \beta = (\textbf{X}^T\textbf{X})^{-1}\textbf{X}^Ty = \beta + (\textbf{X}^T\textbf{X})^{-1}\textbf{X}^T\epsilon\), we have \(E_{\mathcal{T}}(\hat \beta) = \beta\) and

\[\begin{equation} \label{eq:2-5a} \text{Var}_\mathcal{T}(\hat\beta) = E_\mathcal{T}(\hat \beta^T\hat\beta) = E_\mathcal{T}(\textbf{X}^T\textbf{X})^{-1}\sigma^2.\ \ \ \ \ \ \ \end{equation}\]

Note that \(y_0\) is constant for the distribution \(\mathcal{T}\). We have

\[\begin{eqnarray} E_\mathcal{T}(y_0-\hat y_0)^2 &=& y_0^2 + E_\mathcal{T}\hat y_0^2 - 2 y_0E_\mathcal{T}\hat y_0\nonumber\\ &=& [y_0^2-E_{y_0|x_0}(y_0)^2] + [E_\mathcal{T}\hat y_0^2 - (E_\mathcal{T}\hat y_0)^2]\nonumber\\ && + [ (E_\mathcal{T}\hat y_0)^2 - 2 y_0E_\mathcal{T}\hat y_0 + E_{y_0|x_0}(y_0)^2 ]\nonumber\\ &=& [y_0^2-E_{y_0|x_0}(y_0)^2] + [E_\mathcal{T}\hat y_0^2 - (E_\mathcal{T}\hat y_0)^2]\nonumber\\ && + (E_\mathcal{T}\hat y_0 - y_0)^2\nonumber \end{eqnarray}\]

Therefore, by \(\eqref{eq:2-5a}\), we have

\[\begin{eqnarray} E_{y_0|x_0}E_\mathcal{T}(y_0-\hat y_0)^2 &=&\text{Var}(y_0|x_0) + \text{Var}_\mathcal{T}(\hat y_0) + \text{Bias}^2(\hat y_0)\nonumber\\ &=&\sigma^2 + E_\mathcal{T} x_0^T(\textbf{X}^T\textbf{X})^{-1}x_0\sigma^2 + 0^2.\nonumber \end{eqnarray}\]
(b)

First, we have

\[\begin{equation} E_{x_0}\text{EPE}(x_0) \sim E_{x_0}x_0^T\text{Cov}(X)^{-1}x_0\sigma^2/N + \sigma^2.\nonumber \end{equation}\]

Note that \(x_0^T\text{Cov}(X)^{-1}x_0\) is scalar and equal to its own trace, we have

\[\begin{eqnarray} E_{x_0}\text{EPE}(x_0) &\sim& E_{x_0}x_0^T\text{Cov}(X)^{-1}x_0\sigma^2/N + \sigma^2\nonumber\\ &=&\text{trace}\left(E_{x_0}x_0x_0^T\text{Cov}(X)^{-1}\right)\sigma^2/N + \sigma^2\nonumber\\ &=&\text{trace}\left(\textbf{I}_p\right)\sigma^2/N + \sigma^2\nonumber\\ &=&\sigma^2(p/N) + \sigma^2.\nonumber \end{eqnarray}\]