Ex. 3.11

Ex. 3.11

Show that the solution to the multivariate linear regression problem (3.40) is given by (3.39). What happens if the covariance matrices ${\mathbf{\Sigma }}_i$ are different for each observation?

Soln. 3.11

Like (3.38), we write (3.40) in matrix form

$$\text{RSS}(\mathbf{\text{B}};\mathbf{\Sigma })=\text{tr}[(\mathbf{\text{Y}}-\mathbf{\text{X}}\mathbf{\text{B}}{)}^T{\mathbf{\Sigma }}^{-1}(\mathbf{\text{Y}}-\mathbf{\text{X}}\mathbf{\text{B}})].$$

By properties of trace operator, we have

$$\begin{array}{rcl}& & \text{RSS}(\mathbf{\text{B}};\mathbf{\Sigma })\\ & =& \text{tr}[({\mathbf{\text{Y}}}^T{\mathbf{\Sigma }}^{-1}-{\mathbf{\text{B}}}^T{\mathbf{\text{X}}}^T{\mathbf{\Sigma }}^{-1})(\mathbf{\text{Y}}-\mathbf{\text{X}}\mathbf{\text{B}})]\\ & =& \text{tr}({\mathbf{\text{Y}}}^T{\mathbf{\Sigma }}^{-1}\mathbf{\text{Y}}-{\mathbf{\text{Y}}}^T{\mathbf{\Sigma }}^{-1}\mathbf{\text{X}}\mathbf{\text{B}}-{\mathbf{\text{B}}}^T{\mathbf{\text{X}}}^T{\mathbf{\Sigma }}^{-1}\mathbf{\text{Y}}+{\mathbf{\text{B}}}^T{\mathbf{\text{X}}}^T{\mathbf{\Sigma }}^{-1}\mathbf{\text{X}}\mathbf{\text{B}}).\end{array}$$

Taking derivative and setting it to be zero, we get

$$\begin{array}{rcl}& & \frac{\partial \text{RSS}(\mathbf{\text{B}};\mathbf{\Sigma })}{\partial \mathbf{\text{B}}}\\ & =& {\mathbf{\text{X}}}^T({\mathbf{\Sigma }}^{-1}+({\mathbf{\Sigma }}^{-1}{)}^T)\mathbf{\text{X}}\mathbf{\text{B}}-{\mathbf{\text{X}}}^T({\mathbf{\Sigma }}^{-1}+({\mathbf{\Sigma }}^{-1}{)}^T)\mathbf{\text{Y}}\\ \text{(1)}& & =& \mathbf{\text{0}}.\end{array}$$

Note that $\mathbf{\Sigma }$ is a positive definite symmetric matrix, there exists $\mathbf{\text{S}}$ such that ${\mathbf{\Sigma }}^{-1}=\mathbf{\text{S}}{\mathbf{\text{S}}}^T$. Therefore we obtain

$$\begin{array}{rcl}\widehat{\mathbf{\text{B}}}& =& ({\mathbf{\text{X}}}^T\mathbf{\text{S}}{\mathbf{\text{S}}}^T\mathbf{\text{X}}{)}^{-1}{\mathbf{\text{X}}}^T\mathbf{\text{S}}{\mathbf{\text{S}}}^T\mathbf{\text{Y}}\\ & =& ({\mathbf{\text{X}}}^T\mathbf{\text{S}}{\mathbf{\text{S}}}^T\mathbf{\text{X}}{)}^{-1}{\mathbf{\text{X}}}^T\mathbf{\text{S}}{\mathbf{\text{S}}}^T\mathbf{\text{X}}{\mathbf{\text{X}}}^T(\mathbf{\text{X}}{\mathbf{\text{X}}}^T{)}^{-1}\mathbf{\text{Y}}\\ & =& {\mathbf{\text{X}}}^T(\mathbf{\text{X}}{\mathbf{\text{X}}}^T{)}^{-1}\mathbf{\text{Y}}\\ & =& ({\mathbf{\text{X}}}^T\mathbf{\text{X}}{)}^{-1}{\mathbf{\text{X}}}^T\mathbf{\text{X}}{\mathbf{\text{X}}}^T(\mathbf{\text{X}}{\mathbf{\text{X}}}^T{)}^{-1}\mathbf{\text{Y}}\\ & =& ({\mathbf{\text{X}}}^T\mathbf{\text{X}}{)}^{-1}{\mathbf{\text{X}}}^T\mathbf{\text{Y}},\end{array}$$

which is (3.39) in the text.

When ${\mathbf{\Sigma }}_i$ are different, the simple solution for $\mathbf{\text{B}}$ above does not hold. Instead, we have to deal with equations like $\text{(1)}$ with different ${\mathbf{\Sigma }}_i$. Numerical solutions are available though, as the problem is essentially in quadratic form of $\mathbf{\text{B}}$.