Training Deep Nets with Sublinear Memory Cost

Compilers: Principles, Techniques, and Tools

Alfred V. Aho, Ravi Sethi, and Jeffrey D. Ullman · 1986

Earlier work this paper cites.

Long short-term memory

Sepp Hochreiter and Jürgen Schmidhuber · 1997

Earlier work this paper cites.

Algorithm 799: Revolve: An implementation of checkpointing for the reverse or adjoint mode of computational differentiation

Andreas Griewank and Andrea Walther · 2000

Earlier work this paper cites.

Gradient-based learning applied to document recognition

Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner · 2001

Earlier work this paper cites.

Theano: a CPU and GPU math expression compiler

James Bergstra, Olivier Breuleux, Frédéric Bastien, Pascal Lamblin, Razvan Pascanu, Guillaume Desjardins, Joseph Turian, David Warde-Farley, and Yoshua Bengio · 2010

Earlier work this paper cites.

Theano: new features and speed improvements

Frédéric Bastien, Pascal Lamblin, Razvan Pascanu, James Bergstra, Ian J. Goodfellow, Arnaud Bergeron, Nicolas Bouchard, and Yoshua Bengio · 2012

Earlier work this paper cites.

Large scale distributed deep networks

Jeffrey Dean, Greg S. Corrado, Rajat Monga, Kai Chen, Matthieu Devin, Quoc V. Le, Mark Z. Mao, Marc’Aurelio Ranzato, Andrew Senior, Paul Tucker, Ke Yang, and Andrew Y. Ng · 2012

Earlier work this paper cites.

Imagenet classification with deep convolutional neural networks

Alex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton · 2012

Cited alongside, same era.

An introduction to computational networks and the computational network toolkit

Amit Agarwal, Eldar Akchurin, Chris Basoglu, Guoguo Chen, Scott Cyphers, Jasha Droppo, Adam Eversole, Brian Guenter, Mark Hillebrand, Ryan Hoens, Xuedong Huang, Zhiheng Huang, Vladimir Ivanov, Alexey Kamenev, Philipp Kranen, Oleksii Kuchaiev, Wolfgang Manousek, Avner May, Bhaskar Mitra, Olivier Nano, Gaizka Navarro, Alexey Orlov, Marko Padmilac, Hari Parthasarathi, Baolin Peng, Alexey Reznichenko, Frank Seide, Michael L. Seltzer, Malcolm Slaney, Andreas Stolcke, Yongqiang Wang, Huaming Wang, Kaisheng Yao, Dong Yu, Yu Zhang, and Geoffrey Zweig · 2014

Cited alongside, same era.

Long short-term memory recurrent neural network architectures for large scale acoustic modeling

Hasim Sak, Andrew W. Senior, and Françoise Beaufays · 2014

Cited alongside, same era.

TensorFlow: Large-scale machine learning on heterogeneous systems, 2015

Martín Abadi, Ashish Agarwal, Paul Barham, Eugene Brevdo, Zhifeng Chen, Craig Citro, Greg S. Corrado, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Ian Goodfellow, Andrew Harp, Geoffrey Irving, Michael Isard, Yangqing Jia, Rafal Jozefowicz, Lukasz Kaiser, Manjunath Kudlur, Josh Levenberg, Dan Mané, Rajat Monga, Sherry Moore, Derek Murray, Chris Olah, Mike Schuster, Jonathon Shlens, Benoit Steiner, Ilya Sutskever, Kunal Talwar, Paul Tucker, Vincent Vanhoucke, Vijay Vasudevan, Fernanda Viégas, Oriol Vinyals, Pete Warden, Martin Wattenberg, Martin Wicke, Yuan Yu, and Xiaoqiang Zheng · 2015

Batch normalization: Accelerating deep network training by reducing internal covariate shift

Sergey Ioffe and Christian Szegedy · 2015

Later among the works it cites.

Training very deep networks

Original

Rupesh Kumar Srivastava, Klaus Greff, and Jürgen Schmidhuber · 2015

Later among the works it cites.

Highway long short-term memory rnns for distant speech recognition

Original

Yu Zhang, Guoguo Chen, Dong Yu, Kaisheng Yao, Sanjeev Khudanpur, and James Glass · 2015

Later among the works it cites.

Deep learning

Ian Goodfellow, Yoshua Bengio, , and Aaron Courville · 2016

Closest in time.

Identity mappings in deep residual networks

Original

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun · 2016

Closest in time.