Levenberg-Marquardt Optimizer

The Levenberg-Marquardt [96] optimizer (LMMIN) uses a so-called trust region method where the search direction $ \vec{p}$ is a combination of steepest descent and the Newton direction. The search direction is defined by:

$\displaystyle \vec{p}\,({\boldmath J}(\vec{x})^\mathcal{T}{\boldmath J}(\vec{x})+\lambda
{\boldmath I}) = -{\boldmath J}(\vec{x})^\mathcal{T}\vec{f}(\vec{x})$

where $ {\boldmath I}$ denotes the unity matrix, $ \lambda \ge 0$ the Marquardt parameter, and $ {\boldmath J}(\vec{x})$ the Jacobian matrix:

\begin{displaymath}{\boldmath J}(\vec{x}) = \left(
\frac{\partial f_m(\vec{x})}{\partial x_n}

The value of $ \lambda$ is adjusted based on the last evaluation. For a value of $ \lambda=0$ the direction $ \vec{p}$ results in the Newton direction, whereas for $ \lambda=\infty$ the direction $ \vec{p}$ is common to the steepest descent method.

The implementation used in the presented examples is based on the MINPACK project [97,98].