This is the supplementary code (in Python 3) for the paper Y. Malitsky and K. Mishchenko “Adaptive Gradient Descent without Descent” (two-column ICML or one-column arxiv)

The implemented adaptive method is a reliable tool for minimizing differentiable functions. It is among the most general gradient-based algorithms and its fast performance is theoretically guaranteed. The method is merely 2 lines:

There are 5 experiments in total. The first four are provided in the form of a Jupyter notebook and for the neural networks we include a PyTorch implementation of the proposed optimizer.

• Logistic regression
• Matrix factorization
• Cubic regularization
• Linesearch for logisitic regresion
• Neural networks

If you find this code useful, please cite our paper:

@article{malitsky2019adaptive,
  title={Adaptive gradient descent without descent},
  author={Malitsky, Yura and Mishchenko, Konstantin},
  journal={arXiv preprint arXiv:1910.09529},
  year={2019}
}