Ex. 9.4

Suppose the same smoother \(\bb{S}\) is used to estimate both terms in a two-term additive model (i.e., both variables are identical). Assume that \(\bb{S}\) is symmetric with eigenvalues in \([0,1)\). Show that the backfitting residual converges to \((\bb{I} + \bb{S})^{-1}(\bb{I}-\bb{S})\by\), and that the residual sum of squares converges upward. Can the residual sum of squares converge upward in less structured situations? How does this fit compare to the fit with a single term fit by \(\bb{S}\)?

[Hint: Use the eigen-decomposition of \(\bb{S}\) to help with this comparison.]

Soln. 9.4

This follows directly from Ex. 9.2, where the fitted values are both shown to be \((\bb{I}-\bb{S}^2)^{-1}(\bb{S}-\bb{S}^2)\by\). Then, the residual is

\[\begin{eqnarray} \by - 2(\bb{I}-\bb{S}^2)^{-1}(\bb{S}-\bb{S}^2)\by &=&(\bb{I}-\bb{S}^2)^{-1}[(\bb{I}-\bb{S}^2)\by - 2(\bb{S}-\bb{S}^2)\by]\non\\ &=&(\bb{I}-\bb{S}^2)^{-1}(\bb{I} - \bb{S})^2\by\non\\ &=&(\bb{I}+\bb{S})^{-1}(\bb{I}-\bb{S})\by.\non \end{eqnarray}\]

Consider the eigen-decomposition of \(\bb{S}\) (e.g., (5.19) in the text),

\[\begin{equation} \bb{S} = \sum_{k=1}^N\rho_k\bb{u}_k\bb{u}_k^T,\non \end{equation}\]

with \(\rho_k\in [0,1)\). Then the residual can be rewritten as

\[\begin{equation} \sum_{k=1}^N\bb{u}_k\frac{1-\rho_k}{1+\rho_k}\bb{u}_k^T\by.\non \end{equation}\]

If we use a single term fit by \(\bb{S}\), the residual is simply \((\bb{I}-\bb{S})\by\) and is rewritten as

\[\begin{equation} \sum_{k=1}^N\bb{u}_k(1-\rho_k)\bb{u}_k^T\by.\non \end{equation}\]

We see that two-term fit has less residuals compared to one-term fit, which is expected intuitively.