Optimization: what about SGD? about scientific-python-lectures HOT 2 CLOSED

SGD is great, and I use it heavily in machine learning. However, it is a very specific use case, specific in it's input data (statistical data), in it's way of being used, and in it's goal (it doesn't reach very low enery). In addition, it is fragile, as it requires tweaking the "learning rate", which is an open research question.

So, I disagree that it stands in a general-purpose tutorial optimization.

This is an opinion of a machine learning researcher, with a very serious interest in SGD, by the way: I love SGD.

from scientific-python-lectures.

AFAIK, it is proved to converge in convex case if the objective function is the sum of function applied on many let's say, data points. Then, taking derivative wrt some subset of sum elements should work fine.

from scientific-python-lectures.