2017

Neural Optimizer Search with Reinforcement Learning

Bello, Irwan, Zoph, Barret, Vasudevan, Vijay et al.

Understand

We present an approach to automate the process of discovering optimization methods, with a focus on deep learning architectures.

We train a Recurrent Neural Network controller to generate a string in a domain specific language that describes a mathematical update equation based on a list of primitive functions, such as the gradient, running average of the gradient, etc.
The controller is trained with Reinforcement Learning to maximize the performance of a model after a few epochs.
On CIFAR-10, our method discovers several update rules that are better than many commonly used optimizers, such as Adam, RMSProp, or SGD with and without Momentum on a ConvNet model.

Built on

On the limited memory BFGS method for large scale optimization
Liu, Dong C and Nocedal, Jorge · 1989
Earlier work this paper cites.
RPROP - a fast adaptive learning algorithm
Riedmiller, Martin and Braun, Heinrich · 1992
Earlier work this paper cites.
Steps towards ’self-referential’ neural learning: A thought experiment
Schmidhuber, Juergen · 1992
Earlier work this paper cites.
Simple statistical gradient-following algorithms for connectionist reinforcement learning
Williams, Ronald J · 1992
Earlier work this paper cites.
Building a large annotated corpus of english: The penn treebank
Marcus, Mitchell P, Marcinkiewicz, Mary Ann, and Santorini, Beatrice · 1993
Earlier work this paper cites.
Use of genetic programming for the search of a new learning rule for neural networks
Bengio, Samy, Bengio, Yoshua, and Cloutier, Jocelyn · 1994
Earlier work this paper cites.
Long short-term memory
Hochreiter, Sepp and Schmidhuber, Jürgen · 1997
Earlier work this paper cites.
Efficient backprop
LeCun, Yann A, Bottou, Léon, Orr, Genevieve B., and Müller, Klaus-Robert · 1998
Earlier work this paper cites.
Evolution and design of distributed learning rules
Runarsson, Thomas P. and Jonsson, Magnus T · 2000
Earlier work this paper cites.
Learning to learn using gradient descent
Hochreiter, Sepp, Younger, A Steven, and Conwell, Peter R · 2001
Earlier work this paper cites.
Fast curvature matrix-vector products for second-order gradient descent
Schraudolph, Nicol N · 2002
Earlier work this paper cites.
Deep learning via Hessian-free optimization
Martens, James · 2010
Earlier work this paper cites.
Recurrent neural network based language model
Mikolov, Thomas, Karafiat, Martin, Burget, Lukas, Cernocky, Jan, and Sanjeev, Khudanpur · 2010
Earlier work this paper cites.
Adaptive subgradient methods for online learning and stochastic optimization
Duchi, John, Hazan, Elad, and Singer, Yoram · 2011
Earlier work this paper cites.
On optimization methods for deep learning
Le, Quoc V., Ngiam, Jiquan, Coates, Adam, Lahiri, Ahbik, Prochnow, Bobby, and Ng, Andrew Y · 2011
Earlier work this paper cites.
Large scale distributed deep networks
Dean, Jeffrey, Corrado, Greg, Monga, Rajat, Chen, Kai, Devin, Matthieu, Mao, Mark, Senior, Andrew, Tucker, Paul, Yang, Ke, Le, Quoc V, et al · 2012
Earlier work this paper cites.

Similar

Training deep and recurrent networks with Hessian-free optimization
Martens, James and Sutskever, Ilya · 2012
Cited alongside, same era.
Adadelta: an adaptive learning rate method
Original
Zeiler, Matthew D · 2012
Cited alongside, same era.
Revisiting natural gradient for deep networks
Original
Pascanu, Razvan and Bengio, Yoshua · 2013
Cited alongside, same era.
On the difficulty of training recurrent neural networks
Pascanu, Razvan, Mikolov, Tomas, and Bengio, Yoshua · 2013
Cited alongside, same era.
No more pesky learning rates
Schaul, Tom, Zhang, Sixin, and LeCun, Yann · 2013
Cited alongside, same era.
Adam: A method for stochastic optimization
Kingma, Diederik P. and Ba, Jimmy · 2015
Cited alongside, same era.
Tensorflow: A system for large-scale machine learning
Abadi, Martín, Barham, Paul, Chen, Jianmin, Chen, Zhifeng, Davis, Andy, Dean, Jeffrey, Devin, Matthieu, Ghemawat, Sanjay, Irving, Geoffrey, Isard, Michael, Kudlur, Manjunath, Levenberg, Josh, Monga, Rajat, Moore, Sherry, Murray, Derek G., Steiner, Benoit, Tucker, Paul, Vasudevan, Vijay, Warden, Pete, Wicke, Martin, Yu, Yuan, , and Zheng, Xiaoqiang · 2016
Cited alongside, same era.
Learning to learn by gradient descent by gradient descent
Andrychowicz, Marcin, Denil, Misha, Gomez, Sergio, Hoffman, Matthew W., Pfau, David, Schaul, Tom, and de Freitas, Nando · 2016
Cited alongside, same era.
Designing neural network architectures using reinforcement learning
Original
Baker, Bowen, Gupta, Otkrist, Naik, Nikhil, and Raskar, Ramesh · 2016
Cited alongside, same era.
A theoretically grounded application of dropout in recurrent neural networks
Gal, Yarin and Ghahramani, Zoubin · 2016
Cited alongside, same era.
Tying word vectors and word classifiers: A loss framework for language modeling
Inan, Hakan, Khosravi, Khashayar, and Socher, Richard · 2016
Cited alongside, same era.
On large-batch training for deep learning: Generalization gap and sharp minima
Keskar, Nitish Shirish, Mudigere, Dheevatsa, Nocedal, Jorge, Smelyanskiy, Mikhail, and Tang, Ping Tak Peter · 2016
Cited alongside, same era.

Then

Google’s neural machine translation system: Bridging the gap between human and machine translation
Original
Wu, Yonghui, Schuster, Mike, Chen, Zhifeng, Le, Quoc V., Norouzi, Mohammad, Macherey, Wolfgang, Krikun, Maxim, Cao, Yuan, Gao, Qin, Macherey, Klaus, Klingner, Jeff, Shah, Apurva, Johnson, Melvin, Liu, Xiaobing, Kaiser, Lukasz, Gouws, Stephan, Kato, Yoshikiyo, Kudo, Taku, Kazawa, Hideto, Stevens, Keith, Kurian, George, Patil, Nishant, Wang, Wei, Young, Cliff, Smith, Jason, Riesa, Jason, Rudnick, Alex, Vinyals, Oriol, Corrado, Greg, Hughes, Macduff, and Dean, Jeffrey · 2016
Later among the works it cites.
Wide residual networks
Original
Zagoruyko, Sergey and Komodakis, Nikos · 2016
Later among the works it cites.
Distributed second-order optimization using Kronecker-factored approximations
Ba, Jimmy, Grosse, Roger, and Martens, James · 2017
Closest in time.
Google vizier: A service for black-box optimization
Golovin, Daniel, Solnik, Benjamin, Moitra, Subhodeep, Kochanski, Greg, Karro, John Elliot, and Sculley, D · 2017
Closest in time.
Learning to optimize neural nets
Original
Li, Ke and Malik, Jitendra · 2017
Closest in time.
SGDR: stochastic gradient descent with restarts
Loshchilov, Ilya and Hutter, Frank · 2017
Closest in time.
On the state of the art of evaluation in neural language models
Original
Melis, Gabor, Dyer, Chris, and Blunsom, Phil · 2017
Closest in time.
Using the output embedding to improve language models
Original
Press, Ofir and Wolf, Lior · 2017
Closest in time.
Optimization as a model for few-shot learning
Ravi, Sachin and Larochelle, Hugo · 2017
Closest in time.
Proximal policy optimization algorithms
Schulman, John, Wolski, Filip, Dhariwal, Prafulla, Radford, Alec, and Klimov, Oleg · 2017
Closest in time.
Outrageously large neural networks: The sparsely-gated mixture-of-experts layer
Shazeer, Noam, Mirhoseini, Azalia, Maziarz, Krzysztof, Davis, Andy, Le, Quoc, Hinton, Geoffrey, and Dean, Jeff · 2017
Closest in time.
Learned optimizers that scale and generalize
Wichrowska, Olga, Maheswaranathan, Niru, Hoffman, Matthew W., Colmenarejo, Sergio Gomez, Denil, Misha, de Freitas, Nando, and Sohl-Dickstein, Jascha · 2017
Closest in time.
Understanding deep learning requires rethinking generalization
Zhang, Chiyuan, Bengio, Samy, Hardt, Moritz, Recht, Benjamin, and Vinyals, Oriol · 2017
Closest in time.
Neural Architecture Search with reinforcement learning
Zoph, Barret and Le, Quoc V · 2017
Closest in time.
Learning transferable architectures for scalable image recognition
Original
Zoph, Barret, Vasudevan, Vijay, Shlens, Jonathon, and Le, Quoc V · 2017
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…