Adaptive Gradient Methods with Local Guarantees

Understand

Adaptive gradient methods are the method of choice for optimization in machine learning and used to train the largest deep models.

• In this paper we study the problem of learning a local preconditioner, that can change as the data is changing along the optimization trajectory.
• We propose an adaptive gradient method that has provable adaptive regret guarantees vs.
• the best local preconditioner.