MetaGrad: Multiple Learning Rates in Online Learning

Understand

In online convex optimization it is well known that certain subclasses of objective functions are much easier than arbitrary convex functions.

• We are interested in designing adaptive methods that can automatically get fast rates in as many such subclasses as possible, without any manual tuning.
• Previous adaptive methods are able to interpolate between strongly convex and general convex functions.
• We present a new method, MetaGrad, that adapts to a much broader class of functions, including exp-concave and strongly convex functions, but also various types of stochastic and non-stochastic functions without any curvature.