4.1.1 Gradient Based Optimization Methods

The majority of optimization algorithms are based on methods using the gradient and higher derivatives of the target function. These methods approximate the target function $f(\vec{x})$ by a Taylor-series

$$f(\vec{x}_0 + \vec{p}) \approx f(\vec{x}_0) + ({\mathop{\... ...}^{\cal T} {\mathop{\nabla }\nolimits ^2 f(\vec{x}_0)} \vec{p}$$ (4.5)

around $\vec{x}_0$.

From this equation the gradient and, for the second order approximations, the curvature of the target function in the point $\vec{x}_0$ is used to calculate the next step.