Ex. 5.12

Ex. 5.12

Characterize the solution to the following problem,

$$\begin{array}{}\text{(1)}& \underset{f}{min}\text{RSS}(f,\lambda )=\sum _{i=1}^N{\omega }_i\{y_i-f(x_i){\}}^2+\lambda \int \{f^{″}(t{)}^2\}dt,\end{array}$$

where the ${\omega }_i\ge 0$ are observation weights.

Characterize the solution to the smoothing spline problem (5.9) when the training data have ties in $X$.

Soln. 5.12

Following the same arguments in Ex. 5.7, the solution to $\text{(1)}$ is a cubic spline with knots at the unique values of $\{x_i,i=1,...,N\}$ and fitted spline can be represented as

$$\hat{f}(x)=\sum _{j=1}^N{N}_j(x_j){\hat{\theta }}_j$$

where

$$\hat{\theta }=({\mathbf{\text{N}}}^T\mathbf{\text{W}}\mathbf{\text{N}}+\lambda {\mathbf{\Omega }}_N{)}^{-1}{\mathbf{\text{N}}}^T\mathbf{\text{W}}\mathbf{\text{y}}$$

and $\mathbf{\text{W}}=\text{diag}({\omega }_1,...,{\omega }_n)$.

Suppose we group all the training data into $n$ groups. Each group $i$ contains $n_i\ge 1$ training data which have the same $x_i$ and let ${\overline{y}}_i$ be their $y_i$'s average. So the first summand in (5.9) can be rewritten as

$$\begin{array}{rcl}& & \sum _{i=1}^n\sum _{j\in n_i}(y_j-f(x_i){)}^2\\ & =& \sum _{i=1}^n\sum _{j\in n_i}(y_j-{\overline{y}}_i+{\overline{y}}_i-f(x_i){)}^2\\ & =& \sum _{i=1}^n\sum _{j\in n_i}[(y_j-{\overline{y}}_i{)}^2+({\overline{y}}_i-f(x_i){)}^2+2(y_j-{\overline{y}}_i)({\overline{y}}_i-f(x_i))]\end{array}$$

Note that

$$\sum _{i=1}^n\sum _{j\in n_i}({\overline{y}}_i-f(x_i){)}^2=\sum _{i=1}^n{n}_i({\overline{y}}_i-f(x_i){)}^2,$$

and

$$\sum _{i=1}^n\sum _{j\in n_i}(y_j-{\overline{y}}_i)({\overline{y}}_i-f(x_i))=0$$

and $\sum _{i=1}^n\sum _{j\in n_i}(y_j-{\overline{y}}_i{)}^2$ is independent of $f$. So minimizing (5.9) in text is equivalent to minimizing

$$\sum _{i=1}^n{n}_i({\overline{y}}_i-f(x_i){)}^2+\lambda \int {f^{″}(t)}^{2}dt,$$

which is treated by the first half of this problem.