Ex. 6.2

Ex. 6.2

Show that \(\sum_{i=1}^N(x_i-x_0)l_i(x_0) = 0\) for local linear regression. Define \(b_j(x_0) = \sum_{i=1}^N(x_i-x_0)^jl_i(x_0)\). Show that \(b_0(x_0) = 1\) for local polynomial regression of any degree (including local constants). Show that \(b_j(x_0)=0\) for all \(j\in \{1,2,...,k\}\) for local polynomial regression of degree \(k\). What are the implications of this on the bias?

Soln. 6.2

Define the vector-valued function \(b(x)^T = (1,x,x^2,...,x^k)\) for \(k\ge 0\). Let \(\bb{B}\) be the \(N\times (k+1)\) regression matrix with \(i\)th row \(b(x_i)^T\), and \(\bb{W}(x_0)\) the \(N\times N\) diagonal matrix with \(i\)th diagonal element \(K_\lambda(x_0, x_i)\). Then we have

\[\begin{equation} \label{eq:6-3bwb} b(x_0)^T = b(x_0)^T(\bb{B}^T\bb{W}(x_0)\bb{B})^{-1}\bb{B}^T\bb{W}(x_0)\bb{B}. \end{equation}\]

Note the definition of \(l_i(x_0)\) in (6.9) in text, from \(\eqref{eq:6-3bwb}\), we have

\[\begin{eqnarray} 1 &=& b(x_0)^T(\bb{B}^T\bb{W}(x_0)\bb{B})^{-1}\bb{B}^T\bb{W}(x_0)\bb{1} = \sum_{i=1}^Nl_i(x_0)\non\\ x_0 &=& b(x_0)^T(\bb{B}^T\bb{W}(x_0)\bb{B})^{-1}\bb{B}^T\bb{W}(x_0)\bb{B}_2 = \sum_{i=1}^Nl_i(x_0)x_i\non\\ \cdots\non\\ x_0^k &=&b(x_0)^T(\bb{B}^T\bb{W}(x_0)\bb{B})^{-1}\bb{B}^T\bb{W}(x_0)\bb{B}_{k+1} = \sum_{i=1}^Nl_i(x_0)x_i^k\non \end{eqnarray}\]

where \(\bb{B}_i\) is the \(i\)th column of \(\bb{B}\) (note that \(\bb{B}_1=\bb{1}\)).

Therefore we have \(b_0(x_0) = \sum_{i=1}^Nl_i(x_0) = 1\) and

\[\begin{equation} b_1(x_0) = \sum_{i=1}^N(x_i-x_0)l_i(x_0) = \sum_{i=1}^Nl_i(x_0)x_i - x_0\sum_{i=1}^Nl_i(x_0) = x_0 - x_0\cdot 1 = 0.\non \end{equation}\]

For \(j\ge 2\), we have

\[\begin{eqnarray} b_j(x_0) &=& \sum_{i=1}^N(x_i-x_0)^jl_i(x_0)\non\\ &=&\sum_{i=1}^N\left(\sum_{b=0}^jC_j^b(-1)^bx_i^{j-b}x_0^b\right)l_i(x_0)\non\\ &=&\sum_{b=0}^jC_j^b(-1)^bx_0^b\left(\sum_{i=1}^Nl_i(x_0)x_i^{j-b}\right)\non\\ &=&\sum_{b=0}^jC_j^b(-1)^bx_0^bx_0^{j-b}\non\\ &=&\sum_{b=0}^jC_j^b(-1)^bx_0^j\non\\ &=&(1-1)^jx_0^j\non\\ &=& 0.\non \end{eqnarray}\]

By Taylor expansion we have

\[\begin{eqnarray} E[\hat f(x_0)] - f(x_0) &=& \sum_{i=1}^Nl_i(x_0)f(x_i) - f(x_0)\non\\ &=& f(x_0)\sum_{i=1}^Nl_i(x_0) - f(x_0) + f'(x_0)\sum_{i=1}^N(x_i-x_0)l_i(x_0)\non\\ && + \frac{f''(x_0)}{2}\sum_{i=1}^N(x_i-x_0)^2l_i(x_0)\non\\ && + ...\non\\ && + (-1)^k\frac{f^{(k)}}{k!}\sum_{i=1}^N(x_i-x_0)^kl_i(x_0)\non\\ && + R\non\\ &=&R,\non \end{eqnarray}\]

where the remainder term \(R\) involves \((k+1)\)th and higher-order derivatives of \(f\), on which the bias only depends.