Understand
We propose a novel approach to reduce memory consumption of the backpropagation through time (BPTT) algorithm when training recurrent neural networks (RNNs).
- Our approach uses dynamic programming to balance a trade-off between caching of intermediate results and recomputation.
- The algorithm is capable of tightly fitting within almost any user-set memory budget while finding an optimal execution policy minimizing the computational cost.
- Computational devices have limited memory capacity and maximizing a computational performance given a fixed memory budget is a practical use-case.
Built on
Learning internal representations by error propagation
David E Rumelhart, Geoffrey E Hinton, and Ronald J Williams · 1985
Earlier work this paper cites.
Backpropagation through time: what it does and how to do it
Paul J Werbos · 1990
Earlier work this paper cites.
Long short-term memory
Sepp Hochreiter and Jürgen Schmidhuber · 1997
Earlier work this paper cites.
A first look at music composition using LSTM recurrent neural networks
Douglas Eck and Juergen Schmidhuber · 2002
Earlier work this paper cites.
The data-flow equations of checkpointing in reverse automatic differentiation
Benjamin Dauvergne and Laurent Hascoët · 2006
Earlier work this paper cites.
Similar
Generating text with recurrent neural networks
Ilya Sutskever, James Martens, and Geoffrey E Hinton · 2011
Cited alongside, same era.
Supervised Sequence Labelling with Recurrent Neural Networks
Alex Graves · 2012
Cited alongside, same era.
Speech recognition with deep recurrent neural networks
Alan Graves, Abdel-rahman Mohamed, and Geoffrey Hinton · 2013
Cited alongside, same era.
Alex Graves, Greg Wayne, and Ivo Danihelka · 2014
Cited alongside, same era.
Learning to transduce with unbounded memory
Edward Grefenstette, Karl Moritz Hermann, Mustafa Suleyman, and Phil Blunsom · 2015
Cited alongside, same era.
Then
Draw: A recurrent neural network for image generation
Karol Gregor, Ivo Danihelka, Alex Graves, and Daan Wierstra · 2015
Later among the works it cites.
Deep attention recurrent Q-network
Ivan Sorokin, Alexey Seleznev, Mikhail Pavlov, Aleksandr Fedorov, and Anastasiia Ignateva · 2015
Later among the works it cites.
Training deep nets with sublinear memory cost
Tianqi Chen, Bing Xu, Zhiyuan Zhang, and Carlos Guestrin · 2016
Closest in time.
Asynchronous methods for deep reinforcement learning
Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy P Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu · 2016
Closest in time.
Beyond the bibliography
alphaXiv searches the wider corpus for related work and actual follow-ups.
alphaXiv is searching for related work…