Ex. 5.7

Ex. 5.7

Derivation of smoothing splines (Nonparametric Regression and Generalized Linear Models). Suppose that \(N\ge 2\), and that \(g\) is the natural cubic spline interpolant to the pairs \({x_i, z_i}_1^N\), with \(a<x_1<\dots<x_N<b\). This is a natural spline with a knot at every \(x_i\); being an \(N\)-dimensional space of functions, we can determine the coefficients such that it interpolates the sequence \(z_i\) exactly. Let \(\tilde g\) be any other differentiable function on \([a,b]\) that interpolates the \(N\) pairs.

(a)

Let \(h(x) = \tilde g(x) - g(x)\). Use integration by parts and the fact that \(g\) is a natural cubic spline to show that

\[\begin{eqnarray} \int_a^bg''(x)h''(x)dx &=& -\sum_{j=1}^{N-1}g'''(x_j^+)\{h(x_{j+1}-h(x_j))\}\non\\ &=&0.\non \end{eqnarray}\]

(b)

Hence show that

\[\begin{equation} \int_a^b\tilde g''(t)^2dt\ge \int_a^bg''(x)^2dt,\non \end{equation}\]

and the equality can only hold if \(h\) is identically zero in \([a,b]\).

(c)

Consider the penalized least squares problem

\[\begin{equation} \min_f \left[\sum_{i=1}^N(y_i-f(x_i))^2 + \lambda\int_a^bf''(t)^2dt\right].\non \end{equation}\]

Use (b) to argue that the minimizer must be a cubic spline with knots at each of the \(x_i\).

Soln. 5.7

(a)

Using integration by parts, we obtain

\[\begin{eqnarray} \int_a^bg''(x)h''(x)dx &=& [h'(b)g''(b)-h'(a)g''(a)] - \int_a^bg'''(x)h'(x)dx\non\\ &=&0-\int_a^bg'''(x)h'(x)dx\non\\ &=&-\int_{x_1}^{x_N}g'''(x)h'(x)dx,\non \end{eqnarray}\]

where the second equation and the last equation follow from the fact that \(g\) is a natural cubic spline thus linear on \([a, x_1]\) and \([x_N, b]\). Again, since \(g\) is a natural cubic spline, so that \(g'''\) is a constant on each interval \((x_j, x_{j+1})\) for \(j=1,...,N-1\). By definition of integration we obtain the item above is equal to

\[\begin{equation} -\sum_{j=1}^{N-1}g'''(x_j^+)\{h(x_{j+1})-h(x_j)\}.\non \end{equation}\]

Since both \(\tilde g\) and \(g\) interpolate the \(N\) pairs \(\{x_i, z_i\}_1^N\), we know \(h(x_j)=0\) for \(j=1,...,N\). Thus the proof is complete.

(b)

From (a), we have

\[\begin{equation} \int_a^bg''(t)h''(t)dt = \int_a^bg''(t)(\tilde g''(t)-g''(t))dt = 0\non \end{equation}\]

so that

\[\begin{equation} \label{eq:57tmp} \int_a^bg''(t)^2dt = \int_a^b\tilde g''(t)g''(t)dt.\non \end{equation}\]

Then, we start with \(\int_a^b h''(t)^2dt\ge 0\).

\[\begin{eqnarray} \int_a^b h''(t)^2dt &=& \int_a^b (\tilde g''(t)-g''(t))^2dt\non\\ &=&\int_a^b [\tilde g''(t)^2 + g''(t)^2 - 2\tilde g''(t)g''(t)]dt\non\\ &=&\int_a^b [\tilde g''(t)^2 + g''(t)^2 - 2g''(t)^2]dt\non\\ &=&\int_a^b [\tilde g''(t)^2 - g''(t)^2]dt\non\\ &\ge&0. \non \end{eqnarray}\]

Thus we have

\[\begin{equation} \int_a^b\tilde g''(t)^2dt \ge \int_a^bg''(t)^2dt,\non \end{equation}\]

and the equality can only hole if \(h''\) is zero in \([a,b]\). Thus \(h\) is linear in \([a,b]\). However by definition \(h(x_j)=0\) for all \(j=1, ..., N\), we conclude that \(h\) must be identically zero in \([a,b]\).

(c)

Consider a minimizer \(f_0\). We can always construct a natural cubic spline \(f\) such that \(f_0\) and \(f\) have the same values at each of \(\{x_i, i=1,...,N\}\), therefor we have \(\sum_{i=1}^N(y_i-f_0(x_i))^2 = \sum_{i=1}^N(y_i-f(x_i))^2\). From (b) we know that

\[\begin{equation} \int_a^b f_0''(t)^2dt \ge \int_a^b f''(t)^2dt.\non \end{equation}\]

Since \(f_0\) is a minimize, the above inequality is actually a equality, therefore by (b) again, \(f_0=f\) in \([a, b]\). The proof is complete.