3.2.2 Solving the Least-Squares Problem

The response surface must minimize the least-squares of $\vec{\epsilon}$

$$\mathop{\rm min}\limits \vec{\epsilon}^{\cal T} \vec{\epsilon} .$$ (3.7)

Calculation of the first derivatives^3.3 gives

$$\frac{\partial \vec{\epsilon}^{\cal T} \vec{\epsilon}} {\partial \vec{a}} \frac{\partial} {\partial \vec{a}} \left (-\mathcal{Z}\vec{a} + \vec{y}\right )^{\cal T} \left (-\mathcal{Z}\vec{a} + \vec{y}\right ) =$$ = (3.8)

$$\frac{\partial} {\partial \vec{a}} \left (\vec{a}^{\cal T} \mathc... ...vec{a}^{\cal T}\mathcal{Z}^{\cal T}\vec{y} + \vec{y}^{\cal T}\vec{y} \right ) =$$ = (3.9)

$$\left ( \mathcal{Z}^{\cal T} \mathcal{Z} + \left ( \mathcal{Z}^{\... ... (\vec{y}^{\cal T} \mathcal{Z} \right )^{\cal T} - \mathcal{Z}^{\cal T}\vec{y}=$$ = (3.10)

$$2 \mathcal{Z}^{\cal T}\mathcal{Z}\vec{a} - 2 \mathcal{Z}^{\cal T} \vec{y}$$ = (3.11)

Setting (3.11) to zero gives the formula for calculating the response surface

$$\vec{a} = \left ( \mathcal{Z}^{\cal T} \mathcal{Z} \right )^{-1} \mathcal{Z}^{\cal T} \vec{y} .$$ (3.12)

The matrix $\left ( \mathcal{Z}^{\cal T} \mathcal{Z} \right )^{-1} \mathcal{Z}^{\cal T}$ is also called the left-pseudo-inverse. It is a left-inverse of $\mathcal{Z}$ because $\mathcal{Z}$ has to be multiplied by this matrix from the right side to yield the identity matrix $\mathcal{I}$

$$\left ( \mathcal{Z}^{\cal T} \mathcal{Z} \right )^{-1} \mathcal{Z}^{\cal T} \ \mathcal{Z} = \mathcal{I} .$$ (3.13)

For a minimum in least-squares of the error $\vec{\epsilon}$ the second derivative

$$\frac{\partial^2 \vec{\epsilon}^{\cal T} \vec{\epsilon}} {\... ...\cal T} \partial \vec{a}} = \mathcal{Z}^{\cal T} \mathcal{Z}$$ (3.14)

must be nonsingular. This is the case if the if the matrix $\mathcal{Z}$ has full column rank. For practical use the vector $\vec{a}$ can be calculated by a Gauß-solver with the system matrix $\mathcal{Z}^{\cal T} \mathcal{Z}$ and the data vector $\mathcal{Z}^{\cal T} \vec{y}$.

Footnotes

... derivatives^3.3

Some rules for the derivatives of scalar functions with respect to a vector used in (3.10) are given in Appendix B.4.