Ex. 5.12
Ex. 5.12
Characterize the solution to the following problem,
\[\begin{equation}
\label{eq:5-12w}
\min_f\text{RSS}(f,\lambda) = \sum_{i=1}^N\omega_i\{y_i-f(x_i)\}^2 + \lambda\int \{f''(t)^2\}dt,
\end{equation}\]
where the \(\omega_i\ge 0\) are observation weights.
Characterize the solution to the smoothing spline problem (5.9) when the training data have ties in \(X\).
Soln. 5.12
Following the same arguments in Ex. 5.7, the solution to \(\eqref{eq:5-12w}\) is a cubic spline with knots at the unique values of \(\{x_i, i=1,...,N\}\) and fitted spline can be represented as
\[\begin{equation}
\hat f(x) = \sum_{j=1}^NN_j(x_j)\hat \theta_j\non
\end{equation}\]
where
\[\begin{equation}
\hat \theta = (\bb{N}^T\bb{W}\bb{N} + \lambda\bm{\Omega}_N)^{-1}\bb{N}^T\bb{W}\bb{y}\non
\end{equation}\]
and \(\bb{W} = \text{diag}(\omega_1, ..., \omega_n)\).
Suppose we group all the training data into \(n\) groups. Each group \(i\) contains \(n_i\ge 1\) training data which have the same \(x_i\) and let \(\bar y_i\) be their \(y_i\)'s average. So the first summand in (5.9) can be rewritten as
\[\begin{eqnarray}
&&\sum_{i=1}^n\sum_{j\in n_i}(y_j-f(x_i))^2\non\\
&=&\sum_{i=1}^n\sum_{j\in n_i}(y_j-\bar y_i + \bar y_i - f(x_i))^2\non\\
&=&\sum_{i=1}^n\sum_{j\in n_i}[(y_j-\bar y_i)^2 + (\bar y_i - f(x_i))^2 + 2(y_j-\bar y_i)(\bar y_i - f(x_i))]\non
\end{eqnarray}\]
Note that
\[\begin{equation}
\sum_{i=1}^n\sum_{j\in n_i} (\bar y_i - f(x_i))^2 = \sum_{i=1}^n n_i(\bar y_i - f(x_i))^2,\non
\end{equation}\]
and
\[\begin{equation}
\sum_{i=1}^n\sum_{j\in n_i}(y_j-\bar y_i)(\bar y_i - f(x_i)) = 0\non
\end{equation}\]
and \(\sum_{i=1}^n\sum_{j\in n_i}(y_j-\bar y_i)^2\) is independent of \(f\). So minimizing (5.9) in text is equivalent to minimizing
\[\begin{equation}
\sum_{i=1}^nn_i(\bar y_i - f(x_i))^2 + \lambda\int{f''(t)}^2dt,\non
\end{equation}\]
which is treated by the first half of this problem.