On the Impact of the Activation Function on Deep Neural Networks Training

Bayesian Learning for Neural Networks , volume 118

R.M. Neal · 1995

Earlier work this paper cites.

Rectified linear units improve restricted boltzmann machines

V. Nair and G.E. Hinton · 2010

Earlier work this paper cites.

Deep sparse rectifier neural networks

X. Glorot, A. Bordes, and Y. Bengio · 2011

Earlier work this paper cites.

On the number of linear regions of deep neural networks

G.F. Montufar, R. Pascanu, K. Cho, and Y. Bengio · 2014

Earlier work this paper cites.

Empirical evaluation of rectified activations in convolution network

Original

B. Xu, N. Wang, T. Chen, and M. Li · 2015

Earlier work this paper cites.

Delving deep into rectifiers: Surpassing human-level performance on imagenet classification

K. He, X. Zhang, S. Ren, and J. Sun · 2015

Earlier work this paper cites.

Probabilistic backpropagation for scalable learning of Bayesian neural networks

Original

J. M. Hernandez-Lobato and R.P. Adams · 2015

Cited alongside, same era.

Deep Learning

I. Goodfellow, Y. Bengio, and A. Courville · 2016

Cited alongside, same era.

Exponential expressivity in deep neural networks through transient chaos

B. Poole, S. Lahiri, M. Raghu, J. Sohl-Dickstein, and S. Ganguli · 2016

Cited alongside, same era.

Fast and accurate deep network learning by exponential linear units (elus)

D.A. Clevert, T. Unterthiner, and S. Hochreiter · 2016

Cited alongside, same era.

Deep information propagation

S.S. Schoenholz, J. Gilmer, S. Ganguli, and J. Sohl-Dickstein · 2017

Cited alongside, same era.

Searching for activation functions

P. Ramachandran, B. Zoph, and Q.V. Le · 2017

Self-normalizing neural networks

G. Klambauer, T. Unterthiner, and A. Mayr · 2017

Later among the works it cites.

Gaussian process behaviour in wide deep neural networks

A.G. Matthews, J. Hron, M. Rowland, R.E. Turner, and Z. Ghahramani · 2018

Later among the works it cites.

Deep neural networks as Gaussian processes

J. Lee, Y. Bahri, R. Novak, S.S. Schoenholz, J. Pennington, and J. Sohl-Dickstein · 2018

Later among the works it cites.

Comparison of non-linear activation functions for deep neural networks on mnist classification task

D. Pedamonti · 2018

Later among the works it cites.

Expectation propagation: a probabilistic view of deep feed forward networks

Original

M. Milletarí, T. Chotibut, and P. Trevisanutto · 2018

Later among the works it cites.

Scaling limits of wide neural networks with weight sharing: Gaussian process behavior, gradient independence, and neural tangent kernel derivation