2019

Optimizing Millions of Hyperparameters by Implicit Differentiation

Lorraine, Jonathan, Vicol, Paul, Duvenaud, David

Understand

We propose an algorithm for inexpensive gradient-based hyperparameter optimization that combines the implicit function theorem (IFT) with efficient inverse Hessian approximations.

We present results about the relationship between the IFT and differentiating through optimization, motivating our algorithm.
We use the proposed approach to train modern network architectures with millions of weights and millions of hyper-parameters.
For example, we learn a data-augmentation network - where every weight is a hyperparameter tuned for validation performance - outputting augmented training examples.

Built on

Sur les opérations dans les ensembles abstraits et leur application aux équations intégrales
Stefan Banach · 1922
Earlier work this paper cites.
On Bayesian methods for seeking the extremum
Jonas Močkus · 1975
Earlier work this paper cites.
Hedonic housing prices and the demand for clean air
David Harrison Jr and Daniel L Rubinfeld · 1978
Earlier work this paper cites.
Evolutionary principles in self-referential learning, or on learning how to learn: The meta-meta-… hook
Jürgen Schmidhuber · 1987
Earlier work this paper cites.
Building a large annotated corpus of English: The Penn Treebank
Mitchell P Marcus, Mary Ann Marcinkiewicz, and Beatrice Santorini · 1993
Earlier work this paper cites.
Design and regularization of neural networks: The optimal use of a validation set
Jan Larsen, Lars Kai Hansen, Claus Svarer, and M Ohlsson · 1996
Earlier work this paper cites.
Long short-term memory
Sepp Hochreiter and Jürgen Schmidhuber · 1997
Earlier work this paper cites.
Gradient-based learning applied to document recognition
Yann LeCun, Léon Bottou, Yoshua Bengio, Patrick Haffner, et al · 1998
Earlier work this paper cites.
Gradient-based optimization of hyperparameters
Yoshua Bengio · 2000
Earlier work this paper cites.
Learning multiple layers of features from tiny images
Alex Krizhevsky · 2009
Earlier work this paper cites.
Generic methods for optimization-based modeling
Justin Domke · 2012
Earlier work this paper cites.
Practical Bayesian optimization of machine learning algorithms
Jasper Snoek, Hugo Larochelle, and Ryan P Adams · 2012
Earlier work this paper cites.
Random search for hyper-parameter optimization
James Bergstra and Yoshua Bengio · 2012
Earlier work this paper cites.
ImageNet classification with deep convolutional neural networks
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton · 2012
Earlier work this paper cites.
Neural networks for machine learning. Lecture 6a. Overview of mini-batch gradient descent
Geoffrey Hinton, Nitish Srivastava, and Kevin Swersky · 2012
Earlier work this paper cites.
Regularization of neural networks using Dropconnect
Li Wan, Matthew Zeiler, Sixin Zhang, Yann Le Cun, and Rob Fergus · 2013
Earlier work this paper cites.
Generative adversarial nets
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio · 2014
Earlier work this paper cites.
Adam: A method for stochastic optimization
Diederik P Kingma and Jimmy Ba · 2014
Earlier work this paper cites.
Gradient-based hyperparameter optimization through reversible learning
Dougal Maclaurin, David Duvenaud, and Ryan Adams · 2015
Earlier work this paper cites.
Bilevel optimization with nonsmooth lower level problems
Peter Ochs, René Ranftl, Thomas Brox, and Thomas Pock · 2015
Earlier work this paper cites.
Optimizing neural networks with Kronecker-factored approximate curvature
James Martens and Roger Grosse · 2015
Earlier work this paper cites.
U-Net: Convolutional networks for biomedical image segmentation
Olaf Ronneberger, Philipp Fischer, and Thomas Brox · 2015
Earlier work this paper cites.
Variational dropout and the local reparameterization trick
Durk P Kingma, Tim Salimans, and Max Welling · 2015
Earlier work this paper cites.
Scalable gradient-based tuning of continuous regularization hyperparameters
Jelena Luketina, Mathias Berglund, Klaus Greff, and Tapani Raiko · 2016
Earlier work this paper cites.

Similar

Hyperparameter optimization with approximate gradient
Fabian Pedregosa · 2016
Cited alongside, same era.
Learning to learn by gradient descent by gradient descent
Marcin Andrychowicz, Misha Denil, Sergio Gomez, Matthew W Hoffman, David Pfau, Tom Schaul, Brendan Shillingford, and Nando De Freitas · 2016
Cited alongside, same era.
DrMAD: Distilling reverse-mode automatic differentiation for optimizing hyperparameters of deep neural networks
Jie Fu, Hongyin Luo, Jiashi Feng, Kian Hsiang Low, and Tat-Seng Chua · 2016
Cited alongside, same era.
Categorical reparameterization with Gumbel-softmax
Original
Eric Jang, Shixiang Gu, and Ben Poole · 2016
Cited alongside, same era.
Non-stochastic best arm identification and hyperparameter optimization
Kevin Jamieson and Ameet Talwalkar · 2016
Cited alongside, same era.
Deep residual learning for image recognition
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun · 2016
Cited alongside, same era.
A theoretically grounded application of dropout in recurrent neural networks
Yarin Gal and Zoubin Ghahramani · 2016
Cited alongside, same era.
Deep Learning
Ian Goodfellow, Yoshua Bengio, and Aaron Courville · 2016
Cited alongside, same era.
Forward and reverse gradient-based hyperparameter optimization
Luca Franceschi, Michele Donini, Paolo Frasconi, and Massimiliano Pontil · 2017
Cited alongside, same era.
Model-agnostic meta-learning for fast adaptation of deep networks
Chelsea Finn, Pieter Abbeel, and Sergey Levine · 2017
Cited alongside, same era.
Regularization for deep learning: A taxonomy
Original
Jan Kukačka, Vladimir Golkov, and Daniel Cremers · 2017
Cited alongside, same era.
On optimal generalizability in parametric learning
Ahmad Beirami, Meisam Razaviyayn, Shahin Shahrampour, and Vahid Tarokh · 2017
Cited alongside, same era.

Then

Reviving and improving recurrent back-propagation
Renjie Liao, Yuwen Xiong, Ethan Fetaya, Lisa Zhang, KiJung Yoon, Xaq Pitkow, Raquel Urtasun, and Richard Zemel · 2018
Later among the works it cites.
Stochastic hyperparameter optimization through hypernetworks
Original
Jonathan Lorraine and David Duvenaud · 2018
Later among the works it cites.
Metareg: Towards domain generalization using meta-regularization
Yogesh Balaji, Swami Sankaranarayanan, and Rama Chellappa · 2018
Later among the works it cites.
Regularization learning networks: Deep learning for tabular datasets
Ira Shavitt and Eran Segal · 2018
Later among the works it cites.
On first-order meta-learning algorithms
Original
Alex Nichol, Joshua Achiam, and John Schulman · 2018
Later among the works it cites.
Understanding short-horizon bias in stochastic meta-optimization
Original
Yuhuai Wu, Mengye Ren, Renjie Liao, and Roger Grosse · 2018
Later among the works it cites.
The mechanics of n-player differentiable games
David Balduzzi, Sebastien Racaniere, James Martens, Jakob Foerster, Karl Tuyls, and Thore Graepel · 2018
Later among the works it cites.
Parallel architecture and hyperparameter search via successive halving and classification
Original
Manoj Kumar, George E Dahl, Vijay Vasudevan, and Mohammad Norouzi · 2018
Later among the works it cites.
Noisy natural gradient as variational inference
Guodong Zhang, Shengyang Sun, David Duvenaud, and Roger Grosse · 2018
Later among the works it cites.
Learning with opponent-learning awareness
Jakob Foerster, Richard Y Chen, Maruan Al-Shedivat, Shimon Whiteson, Pieter Abbeel, and Igor Mordatch · 2018
Later among the works it cites.
Stable opponent shaping in differentiable games
Original
Alistair Letcher, Jakob Foerster, David Balduzzi, Tim Rocktäschel, and Shimon Whiteson · 2018
Later among the works it cites.
SMASH: One-Shot Model Architecture Search through HyperNetworks
Andrew Brock, Theodore Lim, James Millar Ritchie, and Nicholas J Weston · 2018
Later among the works it cites.
Regularizing and optimizing LSTM language models
Stephen Merity, Nitish Shirish Keskar, and Richard Socher · 2018
Later among the works it cites.
Truncated back-propagation for bilevel optimization
Amirreza Shaban, Ching-An Cheng, Nathan Hatch, and Byron Boots · 2019
Closest in time.
Meta-learning with implicit gradients
Original
Aravind Rajeswaran, Chelsea Finn, Sham Kakade, and Sergey Levine · 2019
Closest in time.
Generalized inner loop meta-learning
Original
Edward Grefenstette, Brandon Amos, Denis Yarats, Phu Mon Htut, Artem Molchanov, Franziska Meier, Douwe Kiela, Kyunghyun Cho, and Soumith Chintala · 2019
Closest in time.
AutoAssist: A framework to accelerate training of deep neural networks
Original
Jiong Zhang, Hsiang-fu Yu, and Inderjit S Dhillon · 2019
Closest in time.
Unsupervised data augmentation
Original
Qizhe Xie, Zihang Dai, Eduard Hovy, Minh-Thang Luong, and Quoc V Le · 2019
Closest in time.
Tuning hyperparameters without grad students: Scalable and robust Bayesian optimisation with Dragonfly
Original
Kirthevasan Kandasamy, Karun Raju Vysyaraju, Willie Neiswanger, Biswajit Paria, Christopher R Collins, Jeff Schneider, Barnabas Poczos, and Eric P Xing · 2019
Closest in time.
Self-tuning networks: Bilevel optimization of hyperparameters using structured best-response functions
Matthew MacKay, Paul Vicol, Jon Lorraine, David Duvenaud, and Roger Grosse · 2019
Closest in time.
On solving minimax optimization locally: A follow-the-ridge approach
Anonymous · 2019
Closest in time.
Deep equilibrium models
Original
Shaojie Bai, J Zico Kolter, and Vladlen Koltun · 2019
Closest in time.
Adversarial learning of general transformations for data augmentation
Saypraseuth Mounsaveng, David Vazquez, Ismail Ben Ayed, and Marco Pedersoli · 2019
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…