Soln. 5.14
Consider
\[\begin{equation}
f(x,y) = \beta_0 + \beta^T(x,y) + \sum_{j=1}^N\alpha_jh_j(x,y)\non
\end{equation}\]
where
\[\begin{eqnarray}
h_j(x,y) &=& [(x-x_j)^2 + (y-y_j)^2]\log\left(\sqrt{(x-x_j)^2 + (y-y_j)^2}\right)\non\\
&=&\frac{1}{2}[(x-x_j)^2 + (y-y_j)^2]\log((x-x_j)^2 + (y-y_j)^2).\non
\end{eqnarray}\]
Without loss of generality, we will drop the constant and linear term \(\beta_0 + \beta^T(x,y)\) in \(f\) since they vanish after taking second derivatives below.
The penalty function \(J\) is
\[\begin{equation}
J(f) = \int\int_{\mathbb{R}^2}\left[\left(\frac{\partial^2f(x,y)}{\partial x^2}\right)^2+2\left(\frac{\partial^2f(x,y)}{\partial x\partial y}\right)^2 + \left(\frac{\partial^2f(x,y)}{\partial y^2}\right)^2\right]dxdy.\non
\end{equation}\]
Next we compute each integrand above. First, denote
\[\begin{eqnarray}
r_{jx} &=& x-x_j\non\\
r_{jy} &=& y-y_j\non\\
r^2_j &=& r^2_{jx} + r^2_{jy}=(x-x_j)^2 + (y-y_j)^2.\non
\end{eqnarray}\]
Then we have
\[\begin{eqnarray}
\frac{\partial f(x,y)}{\partial x} &=& \sum_{j=1}^N\alpha_j[r_{jx}(\log(r_j^2) + 1)]\non\\
\frac{\partial^2 f(x,y)}{\partial^2 x} &=& \sum_{j=1}^N\alpha_j\left[\log(r_j^2) + 2\frac{r_{jx}^2}{r_j^2} + 1\right]\non\\
\frac{\partial^2 f(x,y)}{\partial x\partial y} &=& \sum_{j=1}^N\alpha_j\frac{2r_{jx}r_{jy}}{r_j^2}\non\\
\frac{\partial^2 f(x,y)}{\partial^2 y} &=& \sum_{j=1}^N\alpha_j\left[\log(r_j^2) + 2\frac{r_{jy}^2}{r_j^2} + 1\right].\non
\end{eqnarray}\]
To get penalty \(J[f]\), we calculate the first integrand as
\[\begin{eqnarray}
&&\left(\frac{\partial^2 f(x,y)}{\partial^2 x}\right)^2\non\\
&=&\left(\sum_{j=1}^N\alpha_j\log(r_j^2)\right)^2 + \left(\sum_{j=1}^N2\alpha_j\frac{r_{jx}^2}{r_j^2}\right)^2 + \left(\sum_{j=1}^N\alpha_j\right)^2\non\\
&&+2\left(\sum_{j=1}^N\alpha_j\log(r_j^2)\right)\left(\sum_{j=1}^N2\alpha_j\frac{r_{jx}^2}{r_j^2}\right)\non\\
&&+2\left(\sum_{j=1}^N\alpha_j\log(r_j^2)\right)\left(\sum_{j=1}^N\alpha_j\right)\non\\
&&+2\left(\sum_{j=1}^N2\alpha_j\frac{r_{jx}^2}{r_j^2}\right)\left(\sum_{j=1}^N\alpha_j\right).\non
\end{eqnarray}\]
At this point we see that \(\sum_{j=1}^N\alpha_j = 0\), otherwise the integral would be infinite. So that above integrand is simplified to
\[\begin{eqnarray}
\label{eq:5-14a}
&&\left(\frac{\partial^2 f(x,y)}{\partial^2 x}\right)^2\non\\
&=&\left(\sum_{j=1}^N\alpha_j\log(r_j^2)\right)^2 + \left(\sum_{j=1}^N2\alpha_j\frac{r_{jx}^2}{r_j^2}\right)^2+2\left(\sum_{j=1}^N\alpha_j\log(r_j^2)\right)\left(\sum_{j=1}^N2\alpha_j\frac{r_{jx}^2}{r_j^2}\right).\non
\end{eqnarray}\]
Similarly we have
\[\begin{eqnarray}
\label{eq:5-14b}
\left(\frac{\partial^2 f(x,y)}{\partial x\partial y}\right)^2=\left(\sum_{j=1}^N\alpha_j\frac{2r_{jx}r_{jy}}{r_j^2}\right)^2.\non
\end{eqnarray}\]
and
\[\begin{eqnarray}
\label{eq:5-14c}
&&\left(\frac{\partial^2 f(x,y)}{\partial^2 y}\right)^2\non\\
&=&\left(\sum_{j=1}^N\alpha_j\log(r_j^2)\right)^2 + \left(\sum_{j=1}^N2\alpha_j\frac{r_{jy}^2}{r_j^2}\right)^2+2\left(\sum_{j=1}^N\alpha_j\log(r_j^2)\right)\left(\sum_{j=1}^N2\alpha_j\frac{r_{jy}^2}{r_j^2}\right).\non
\end{eqnarray}\]
Sum them up we get
\[\begin{eqnarray}
&&2\left(\sum_{j=1}^N\alpha_j\log(r_j^2)\right)^2 + \left(\sum_{j=1}^N2\alpha_j\frac{r_{jx}^2}{r_j^2}\right)^2+\left(\sum_{j=1}^N2\alpha_j\frac{r_{jy}^2}{r_j^2}\right)^2\non\\
&& + \left(\sum_{j=1}^N\alpha_j\frac{2r_{jx}r_{jy}}{r_j^2}\right)^2\non\\
&& + 2\left(\sum_{j=1}^N\alpha_j\log(r_j^2)\right)\left(\sum_{j=1}^N2\alpha_j\frac{r_{jx}^2+r_{jy}^2}{r_j^2}\right).\non
\end{eqnarray}\]
Note that \(\frac{r_{jx}^2+r_{jy}^2}{r_j^2}=1\) and \(\sum_{j=1}^N\alpha_j=0\), the last summand above vanishes and we are left with
\[\begin{equation}
\left(\sum_{j=1}^N\alpha_j\log(r_j^2)\right)^2 + \left(\sum_{j=1}^N2\alpha_j\frac{r_{jx}^2}{r_j^2}\right)^2+\left(\sum_{j=1}^N2\alpha_j\frac{r_{jy}^2}{r_j^2}\right)^2+ \left(\sum_{j=1}^N\alpha_j\frac{2r_{jx}r_{jy}}{r_j^2}\right)^2.\non
\end{equation}\]
Remark
It has been shown that \(\sum_{j=1}^N\alpha_jx_j=0\) (see, e.g., Thin-Plate Splines) , however I don't see how to arrive that from here.