This is an implementation of Gradient Descent algorithm frequently encountered in Machine Learning/Neural Networks. This implementation is heavily influenced by QuantitativeBytes implementation, I have enhanced the original by using C++17 lambdas instead of function pointers and replacing finite difference derivatives with Automatic Algorithmic Differentiation using AutoDiff

Gradient Descent is a numerical technique for finding the minimum of a function. For example, consider function as shown below. We know that analytically the first derivative is

Starting with point , the equation to compute the next point is :

where is a constant, called Step Size or Learning Rate.

To compute , either use finite difference which will give you an approximate derivative of the function as follows :

or use AAD which will give you exact derivatives.

Finally, here are the test results for finding the minima of the three example functions.

As you can see, using AAD, you get a value much closer to zero when zero is the function minimum.