2019

The Role of Neural Network Activation Functions

Parhi, Rahul, Nowak, Robert D.

Understand

A wide variety of activation functions have been proposed for neural networks.

The Rectified Linear Unit (ReLU) is especially popular today.
There are many practical reasons that motivate the use of the ReLU.
This paper provides new theoretical characterizations that support the use of the ReLU, its variants such as the leaky ReLU, as well as other activation functions in the case of univariate, single-hidden layer feedforward neural networks.

Built on

G. Cybenko, “Approximation by superpositions of a sigmoidal function,” Mathematics of control, signals and systems , vol. 2, no. 4, pp. 303–314, 1989
1989
Earlier work this paper cites.
K. Hornik, M. Stinchcombe, and H. White, “Multilayer feedforward networks are universal approximators,” Neural networks , vol. 2, no. 5, pp. 359–366, 1989
1989
Earlier work this paper cites.
K.-I. Funahashi, “On the approximate realization of continuous mappings by neural networks,” Neural networks , vol. 2, no. 3, pp. 183–192, 1989
1989
Earlier work this paper cites.
A. Krogh and J. A. Hertz, “A simple weight decay can improve generalization,” in Advances in neural information processing systems , 1992, pp. 950–957
1992
Earlier work this paper cites.
J. E. Moody, “The effective number of parameters: An analysis of generalization and regularization in nonlinear learning systems,” in Advances in neural information processing systems , 1992, pp. 847–854
1992
Earlier work this paper cites.
A. R. Barron, “Universal approximation bounds for superpositions of a sigmoidal function,” IEEE Transactions on Information theory , vol. 39, no. 3, pp. 930–945, 1993
1993
Earlier work this paper cites.
M. Leshno, V. Y. Lin, A. Pinkus, and S. Schocken, “Multilayer feedforward networks with a nonpolynomial activation function can approximate any function,” Neural networks , vol. 6, no. 6, pp. 861–867, 1993
1993
Earlier work this paper cites.
M. Unser and T. Blu, “Fractional splines and wavelets,” SIAM review , vol. 42, no. 1, pp. 43–67, 2000
2000
Earlier work this paper cites.
X. Glorot, A. Bordes, and Y. Bengio, “Deep sparse rectifier neural networks,” in Proceedings of the fourteenth international conference on artificial intelligence and statistics , 2011, pp. 315–323
2011
Earlier work this paper cites.

Similar

J. Duchi, E. Hazan, and Y. Singer, “Adaptive subgradient methods for online learning and stochastic optimization,” Journal of Machine Learning Research , vol. 12, no. Jul, pp. 2121–2159, 2011
2011
Cited alongside, same era.
A. L. Maas, A. Y. Hannun, and A. Y. Ng, “Rectifier nonlinearities improve neural network acoustic models,” in ICML Workshop on Deep Learning for Audio, Speech and Language Processing , 2013
2013
Cited alongside, same era.
B. Neyshabur, R. R. Salakhutdinov, and N. Srebro, “Path-SGD: Path-normalized optimization in deep neural networks,” in Advances in Neural Information Processing Systems , 2015, pp. 2422–2430
2015
Cited alongside, same era.
Y. LeCun, Y. Bengio, and G. Hinton, “Deep learning,” Nature , vol. 521, no. 7553, pp. 436–444, 2015
2015
Cited alongside, same era.
K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in The IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , June 2016
2016
Cited alongside, same era.
B. Neyshabur, S. Bhojanapalli, D. Mcallester, and N. Srebro, “Exploring generalization in deep learning,” in Advances in Neural Information Processing Systems , 2017, pp. 5947–5956
2017
Cited alongside, same era.
M. Unser, J. Fageot, and J. P. Ward, “Splines are universal solutions of linear inverse problems with generalized TV regularization,” SIAM Review , vol. 59, no. 4, pp. 769–793, 2017
2017
Cited alongside, same era.
J. M. Klusowski and A. R. Barron, “Approximation by combinations of ReLU and squared ReLU ridge functions with ℓ 1 \ell^{1} and ℓ 0 \ell^{0} controls,” IEEE Transactions on Information Theory , vol. 64, no. 12, pp. 7649–7656, 2018
2018
Cited alongside, same era.

Then

H. Gupta, J. Fageot, and M. Unser, “Continuous-domain solutions of linear inverse problems with Tikhonov versus generalized TV regularization,” IEEE Transactions on Signal Processing , vol. 66, no. 17, pp. 4670–4684, 2018
2018
Later among the works it cites.
C. Wei, J. D. Lee, Q. Liu, and T. Ma, “Regularization matters: Generalization and optimization of neural nets vs their induced kernel,” in Advances in Neural Information Processing Systems , 2019, pp. 9709–9721
2019
Closest in time.
T. Debarre, J. Fageot, H. Gupta, and M. Unser, “B-spline-based exact discretization of continuous-domain inverse problems with generalized TV regularization,” IEEE Transactions on Information Theory , 2019
2019
Closest in time.
P. H. P. Savarese, I. Evron, D. Soudry, and N. Srebro, “How do infinite width bounded norm networks look in function space?” in Conference on Learning Theory, COLT 2019, 25-28 June 2019, Phoenix, AZ, USA , 2019, pp. 2667–2690
2019
Closest in time.
M. Unser, “A representer theorem for deep neural networks,” Journal of Machine Learning Research , vol. 20, no. 110, pp. 1–30, 2019
2019
Closest in time.
S. Aziznejad and M. Unser, “Deep spline networks with control of Lipschitz regularity,” in ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) . IEEE, 2019, pp. 3242–3246
2019
Closest in time.
H. N. Mhaskar, “Dimension independent bounds for general shallow networks,” Neural Networks , vol. 123, pp. 142–152, 2020
2020
Closest in time.
R. Parhi and R. D. Nowak, “Neural networks, ridge splines, and TV regularization in the Radon domain,” arXiv preprint arXiv:2006.05626v1 , 2020
Original
2020
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…