Scalable Nested Optimization for Deep Learning

Understand

Gradient-based optimization has been critical to the success of machine learning, updating a single set of parameters to minimize a single loss.

• A growing number of applications rely on a generalization of this, where we have a bilevel or nested optimization of which subsets of parameters update on different objectives nested inside each other.
• We focus on motivating examples of hyperparameter optimization and generative adversarial networks.
• However, naively applying classical methods often fails when we look at solving these nested problems on a large scale.