Fetching the paper…

Reading the bibliography…

The empirical size of trained neural networks · Around

2016

The empirical size of trained neural networks

Chen, Kevin K., Gamst, Anthony, Walker, Alden

Original
PDF
Post on X

Understand

ReLU neural networks define piecewise linear functions of their inputs.

However, initializing and training a neural network is very different from fitting a linear spline.
In this paper, we expand empirically upon previous theoretical work to demonstrate features of trained neural networks.
Standard network initialization and training produce networks vastly simpler than a naive parameter count would suggest and can impart odd features to the trained network.

Built on

Peter L. Bartlett and Shahar Mendelson. Rademacher and Gaussian Complexities: Risk Bounds and Structural Results . Journal of Machine Learning Research 3
2002
Earlier work this paper cites.
Xavier Glorot and Yoshua Bengio Understanding the difficulty of training deep feedforward neural networks , International Conference on Artificial Intelligence and Statistics, 2010
2010
Earlier work this paper cites.

Similar

Guido Montúfar, Razvan Pascanu, Kyunghyun Cho, and Yoshua Bengio, On the number of linear regions of deep neural networks , Advances in Neural Information Processing Systems 2014, 2924–2932
2014
Cited alongside, same era.
Pascanu Razvan, Guido Montúfar, and Yoshua Bengio, On the number of response regions of deep feedforward networks with piecewise linear activations , preprint: arxiv:1312.6098v5, 2014
Original
2014
Cited alongside, same era.
Kevin K. Chen. The upper bound on knots in neural networks , preprint
Cited in the paper.
Kevin K. Chen, Anthony Gamst, and Alden Walker. Knots in random neural networks , preprint
Cited in the paper.
Kevin K. Chen, Anthony Gamst, and Alden Walker. The empirical size and risk of trained neural networks , preprint
Cited in the paper.

Then

François Chollet. Keras . Github, 2015. URL: https://github.com/fchollet/keras
2015
Later among the works it cites.
Maithra Raghu, Ben Poole, Jon Kleinberg, Surya Ganguli, and Jascha Sohl-Dickstein, On the expressive power of deep neural networks , preprint: arXiv:1606.05336v2, 2016
Original
2016
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…