A general system of differential equations to model first order adaptive algorithms

Understand

First order optimization algorithms play a major role in large scale machine learning.

• A new class of methods, called adaptive algorithms, were recently introduced to adjust iteratively the learning rate for each coordinate.
• Despite great practical success in deep learning, their behavior and performance on more general loss functions are not well understood.
• In this paper, we derive a non-autonomous system of differential equations, which is the continuous time limit of adaptive optimization methods.