Mean field theory for deep dropout networks: digging up gradient backpropagation deeply

Convolutional networks for images, speech, and time series

Yann LeCun, Yoshua Bengio, et al · 1995

Earlier work this paper cites.

Priors for infinite networks

Radford M Neal · 1996

Earlier work this paper cites.

Learning to forget: Continual prediction with LSTM

Felix A Gers, Jürgen Schmidhuber, and Fred Cummins · 1999

Earlier work this paper cites.

Recurrent neural network based language model

Tomáš Mikolov, Martin Karafiát, Lukáš Burget, Jan Černockỳ, and Sanjeev Khudanpur · 2010

Earlier work this paper cites.

Dropout: a simple way to prevent neural networks from overfitting

Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov · 2014

Earlier work this paper cites.

Empirical evaluation of gated recurrent neural networks on sequence modeling

Original

Junyoung Chung, Caglar Gulcehre, KyungHyun Cho, and Yoshua Bengio · 2014

Earlier work this paper cites.

Deep learning

Yann LeCun, Yoshua Bengio, and Geoffrey Hinton · 2015

Earlier work this paper cites.

Batch normalization: Accelerating deep network training by reducing internal covariate shift

Original

Sergey Ioffe and Christian Szegedy · 2015

Earlier work this paper cites.

Deep residual learning for image recognition

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun · 2016

Earlier work this paper cites.

Exponential expressivity in deep neural networks through transient chaos

Ben Poole, Subhaneil Lahiri, Maithra Raghu, Jascha Sohl-Dickstein, and Surya Ganguli · 2016

Cited alongside, same era.

Deep information propagation

Original

Samuel S Schoenholz, Justin Gilmer, Surya Ganguli, and Jascha Sohl-Dickstein · 2016

Cited alongside, same era.

Toward deeper understanding of neural networks: The power of initialization and a dual view on expressivity

Amit Daniely, Roy Frostig, and Yoram Singer · 2016

Cited alongside, same era.

Dropout as a Bayesian approximation: Representing model uncertainty in deep learning

Original

Yarin Gal and Zoubin Ghahramani · 2016

Cited alongside, same era.

Resurrecting the sigmoid in deep learning through dynamical isometry: theory and practice

Jeffrey Pennington, Samuel Schoenholz, and Surya Ganguli · 2017

Cited alongside, same era.

Dynamical isometry and a mean field theory of RNNs: Gating enables signal propagation in recurrent neural networks

Original

Minmin Chen, Jeffrey Pennington, and Samuel S Schoenholz · 2018

Later among the works it cites.

The emergence of spectral universality in deep networks

Original

Jeffrey Pennington, Samuel S Schoenholz, and Surya Ganguli · 2018

Later among the works it cites.

Gaussian process behaviour in wide deep neural networks

Original

Alexander G de G Matthews, Mark Rowland, Jiri Hron, Richard E Turner, and Zoubin Ghahramani · 2018

Later among the works it cites.

Neural tangent kernel: Convergence and generalization in neural networks

Arthur Jacot, Franck Gabriel, and Clément Hongler · 2018

Later among the works it cites.

Scaling limits of wide neural networks with weight sharing: Gaussian process behavior, gradient independence, and neural tangent kernel derivation