A Nonstochastic Control Approach to Optimization

Understand

Selecting the best hyperparameters for a particular optimization instance, such as the learning rate and momentum, is an important but nonconvex problem.

• As a result, iterative optimization methods such as hypergradient descent lack global optimality guarantees in general.
• We propose an online nonstochastic control methodology for mathematical optimization.
• First, we formalize the setting of meta-optimization, an online learning formulation of learning the best optimization algorithm from a class of methods.