Eve: A Gradient Based Optimization Method with Locally and Globally Adaptive Learning Rates

Understand

Adaptive gradient methods for stochastic optimization adjust the learning rate for each parameter locally.

• However, there is also a global learning rate which must be tuned in order to get the best performance.
• In this paper, we present a new algorithm that adapts the learning rate locally for each parameter separately, and also globally for all parameters together.
• Specifically, we modify Adam, a popular method for training deep learning models, with a coefficient that captures properties of the objective function.