Deep Online Convex Optimization by Putting Forecaster to Sleep

Understand

Methods from convex optimization such as accelerated gradient descent are widely used as building blocks for deep learning algorithms.

• However, the reasons for their empirical success are unclear, since neural networks are not convex and standard guarantees do not apply.
• This paper develops the first rigorous link between online convex optimization and error backpropagation on convolutional networks.
• The first step is to introduce circadian games, a mild generalization of convex games with similar convergence properties.