Ex. 14.18

Consider the regularized log-likelihood for the density estimation problem arising in ICA,

\[\begin{equation} \frac{1}{N}\sum_{i=1}^N[\log(\phi(s_i))+g(s_i)] - \int \phi(t)e^{g(t)}dt - \lambda \int \{g^{'''}(t)\}^2dt.\non \end{equation}\]

The solution \(\hat g\) is a quartic smoothing spline, and can be written as \(\hat g(s) = \hat q(s) + \hat{q}_\bot (s)\), where q is a quadratic function (in the null space of the penalty). Let \(q(s) = \theta_0 + \theta_1s + \theta_2s^2\). By examining the stationarity conditions for \(\theta^k\), \(k = 1, 2, 3\), show that the solution \(\hat f = \phi e^{\hat g}\) is a density, and has mean zero and variance one. If we used a second-derivative penalty \(\int \{g^{''}(t)\}^2(t)dt\) instead, what simple modification could we make to the problem to maintain the three moment conditions?

Soln. 14.18

Considering the stationarity condition, if we differentiate the log-likelihood w.r.t to \(\theta_0\), we should get zero at the optimal value \(\hat\theta_0\). By the assumption of \(\hat g(s)\), the differential is

\[\begin{equation} 1 - \int \phi(t)e^{\hat g(t)} dt = 0,\non \end{equation}\]

therefore we know \(\hat f= \phi e^{\hat g}\) is a density.

Similarly for \(\theta_1\), the differential is

\[\begin{equation} \bar s - \int \phi(t)e^{\hat g(t)}tdt=0-\text{mean}(\hat f)=0,\non \end{equation}\]

since \(S\) is assumed to have zero mean.

At last, for \(\theta_2\), the differential is

\[\begin{equation} \frac{1}{N}\sum_{i=1}^Ns_i^2 - \int \phi(t)e^{\hat g(t)}t^2dt=0.\non \end{equation}\]

Since both \(S\) and \(\hat f\) have zero mean, we know that the variance of \(\hat f\) equals the variance of \(S\), which is assumed to be 1.

If we use a second-derivative penalty instead, then \(\hat g\) becomes a cubic smoothing spline, and the first two conditions (\(\hat f\) is a density and has zero mean) are still preserved via the same arguments above. We can modify the problem by adding a Lagrangian multiplier to accommodate the constraint for the variance.