Built on

S. Hayou, A. Doucet, and J. Rousseau, “On the impact of the activation function on deep neural networks training,” in Proceedings of the 36th International Conference on Machine Learning , Proceedings of Machine Learning Research, Vol. 97, edited by K. Chaudhuri and R. Salakhutdinov (PMLR, 2019) pp. 2672–2680, arXiv:1902.06853 [stat.ML]
Original
1902
Earlier work this paper cites.
T. B. Brown, B. Mann, N. Ryder, M. Subbiah, J. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, S. Agarwal, A. Herbert-Voss, G. Krueger, T. Henighan, R. Child, A. Ramesh, D. M. Ziegler, J. Wu, C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, S. Gray, B. Chess, J. Clark, C. Berner, S. McCandlish, A. Radford, I. Sutskever, and D. Amodei, “Language Models are Few-Shot Learners,” arXiv e-prints , arXiv:2005.14165 (2020), arXiv:2005.14165 [cs.CL]
Original
2005
Earlier work this paper cites.
D. Hernandez and T. B. Brown, “Measuring the Algorithmic Efficiency of Neural Networks,” arXiv e-prints , arXiv:2005.04305 (2020), arXiv:2005.04305 [cs.LG]
Original
2005
Earlier work this paper cites.
A. Krizhevsky, I. Sutskever, and G. E. Hinton, “Imagenet classification with deep convolutional neural networks,” in Advances in Neural Information Processing Systems , Vol. 25, edited by F. Pereira, C. Burges, L. Bottou, and K. Weinberger (Curran Associates, Inc., 2012)
2012
Earlier work this paper cites.
R. Livni, S. Shalev-Shwartz, and O. Shamir, “On the Computational Efficiency of Training Neural Networks,” arXiv e-prints , arXiv:1410.1141 (2014), arXiv:1410.1141 [cs.LG]
Original
2014
Earlier work this paper cites.
M. Raghu, B. Poole, J. Kleinberg, S. Ganguli, and J. Sohl-Dickstein, “On the Expressive Power of Deep Neural Networks,” arXiv e-prints , arXiv:1606.05336 (2016), arXiv:1606.05336 [stat.ML]
Original
2016
Earlier work this paper cites.
B. Poole, S. Lahiri, M. Raghu, J. Sohl-Dickstein, and S. Ganguli, “Exponential expressivity in deep neural networks through transient chaos,” (2016), 10.48550/arXiv.1606.05340
2016
Earlier work this paper cites.

Similar

S. S. Schoenholz, J. Gilmer, S. Ganguli, and J. Sohl-Dickstein, “Deep Information Propagation,” arXiv e-prints , arXiv:1611.01232 (2016), arXiv:1611.01232 [stat.ML]
Original
2016
Cited alongside, same era.
L. Xiao, Y. Bahri, J. Sohl-Dickstein, S. S. Schoenholz, and J. Pennington, “Dynamical isometry and a mean field theory of cnns: How to train 10,000-layer vanilla convolutional neural networks,” (2018), 10.48550/arXiv.1806.05393
2018
Cited alongside, same era.
M. Chen, J. Pennington, and S. S. Schoenholz, “Dynamical isometry and a mean field theory of rnns: Gating enables signal propagation in recurrent neural networks,” (2018), 10.48550/arXiv.1806.05394
2018
Cited alongside, same era.
S. Hayou, A. Doucet, and J. Rousseau, “On the Selection of Initialization and Activation Function for Deep Neural Networks,” arXiv e-prints , arXiv:1805.08266 (2018), arXiv:1805.08266 [stat.ML]
Original
2018
Cited alongside, same era.

Then

A. Ramesh, M. Pavlov, G. Goh, S. Gray, C. Voss, A. Radford, M. Chen, and I. Sutskever, “Zero-Shot Text-to-Image Generation,” arXiv e-prints , arXiv:2102.12092 (2021), arXiv:2102.12092 [cs.CV]
Original
2021
Later among the works it cites.
J. Jumper, R. Evans, A. Pritzel, T. Green, M. Figurnov, O. Ronneberger, K. Tunyasuvunakool, R. Bates, A. Žídek, A. Potapenko, A. Bridgland, C. Meyer, S. A. A. Kohl, A. J. Ballard, A. Cowie, B. Romera-Paredes, S. Nikolov, R. Jain, J. Adler, T. Back, S. Petersen, D. Reiman, E. Clancy, M. Zielinski, M. Steinegger, M. Pacholska, T. Berghammer, S. Bodenstein, D. Silver, O. Vinyals, A. W. Senior, K. Kavukcuoglu, P. Kohli, and D. Hassabis, “Highly accurate protein structure prediction with alphafold,” Nature 596
2021
Later among the works it cites.
D. A. Roberts, S. Yaida, and B. Hanin, “The Principles of Deep Learning Theory,” (2021), 10.1017/9781009023405 , arXiv:2106.10165 [cs.LG]
Original
2021
Later among the works it cites.
J. Erdmenger, K. T. Grosvenor, and R. Jefferson, “Towards quantifying information flows: relative entropy in deep neural networks and the renormalization group,” SciPost Phys. 12

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…

2023

Criticality versus uniformity in deep neural networks

Bukva, Aleksandar, de Gier, Jurriaan, Grosvenor, Kevin T. et al.

Understand

Deep feedforward networks initialized along the edge of chaos exhibit exponentially superior training ability as quantified by maximum trainable depth.

In this work, we explore the effect of saturation of the tanh activation function along the edge of chaos.
In particular, we determine the line of uniformity in phase space along which the post-activation distribution has maximum entropy.
This line intersects the edge of chaos, and indicates the regime beyond which saturation of the activation function begins to impede training efficiency.

Built on

S. Hayou, A. Doucet, and J. Rousseau, “On the impact of the activation function on deep neural networks training,” in Proceedings of the 36th International Conference on Machine Learning , Proceedings of Machine Learning Research, Vol. 97, edited by K. Chaudhuri and R. Salakhutdinov (PMLR, 2019) pp. 2672–2680, arXiv:1902.06853 [stat.ML]
Original
1902
Earlier work this paper cites.
T. B. Brown, B. Mann, N. Ryder, M. Subbiah, J. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, S. Agarwal, A. Herbert-Voss, G. Krueger, T. Henighan, R. Child, A. Ramesh, D. M. Ziegler, J. Wu, C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, S. Gray, B. Chess, J. Clark, C. Berner, S. McCandlish, A. Radford, I. Sutskever, and D. Amodei, “Language Models are Few-Shot Learners,” arXiv e-prints , arXiv:2005.14165 (2020), arXiv:2005.14165 [cs.CL]
Original
2005
Earlier work this paper cites.
D. Hernandez and T. B. Brown, “Measuring the Algorithmic Efficiency of Neural Networks,” arXiv e-prints , arXiv:2005.04305 (2020), arXiv:2005.04305 [cs.LG]
Original
2005
Earlier work this paper cites.
A. Krizhevsky, I. Sutskever, and G. E. Hinton, “Imagenet classification with deep convolutional neural networks,” in Advances in Neural Information Processing Systems , Vol. 25, edited by F. Pereira, C. Burges, L. Bottou, and K. Weinberger (Curran Associates, Inc., 2012)
2012
Earlier work this paper cites.
R. Livni, S. Shalev-Shwartz, and O. Shamir, “On the Computational Efficiency of Training Neural Networks,” arXiv e-prints , arXiv:1410.1141 (2014), arXiv:1410.1141 [cs.LG]
Original
2014
Earlier work this paper cites.
M. Raghu, B. Poole, J. Kleinberg, S. Ganguli, and J. Sohl-Dickstein, “On the Expressive Power of Deep Neural Networks,” arXiv e-prints , arXiv:1606.05336 (2016), arXiv:1606.05336 [stat.ML]
Original
2016
Earlier work this paper cites.
B. Poole, S. Lahiri, M. Raghu, J. Sohl-Dickstein, and S. Ganguli, “Exponential expressivity in deep neural networks through transient chaos,” (2016), 10.48550/arXiv.1606.05340
2016
Earlier work this paper cites.

Similar

S. S. Schoenholz, J. Gilmer, S. Ganguli, and J. Sohl-Dickstein, “Deep Information Propagation,” arXiv e-prints , arXiv:1611.01232 (2016), arXiv:1611.01232 [stat.ML]
Original
2016
Cited alongside, same era.
L. Xiao, Y. Bahri, J. Sohl-Dickstein, S. S. Schoenholz, and J. Pennington, “Dynamical isometry and a mean field theory of cnns: How to train 10,000-layer vanilla convolutional neural networks,” (2018), 10.48550/arXiv.1806.05393
2018
Cited alongside, same era.
M. Chen, J. Pennington, and S. S. Schoenholz, “Dynamical isometry and a mean field theory of rnns: Gating enables signal propagation in recurrent neural networks,” (2018), 10.48550/arXiv.1806.05394
2018
Cited alongside, same era.
S. Hayou, A. Doucet, and J. Rousseau, “On the Selection of Initialization and Activation Function for Deep Neural Networks,” arXiv e-prints , arXiv:1805.08266 (2018), arXiv:1805.08266 [stat.ML]
Original
2018
Cited alongside, same era.

Then

A. Ramesh, M. Pavlov, G. Goh, S. Gray, C. Voss, A. Radford, M. Chen, and I. Sutskever, “Zero-Shot Text-to-Image Generation,” arXiv e-prints , arXiv:2102.12092 (2021), arXiv:2102.12092 [cs.CV]
Original
2021
Later among the works it cites.
J. Jumper, R. Evans, A. Pritzel, T. Green, M. Figurnov, O. Ronneberger, K. Tunyasuvunakool, R. Bates, A. Žídek, A. Potapenko, A. Bridgland, C. Meyer, S. A. A. Kohl, A. J. Ballard, A. Cowie, B. Romera-Paredes, S. Nikolov, R. Jain, J. Adler, T. Back, S. Petersen, D. Reiman, E. Clancy, M. Zielinski, M. Steinegger, M. Pacholska, T. Berghammer, S. Bodenstein, D. Silver, O. Vinyals, A. W. Senior, K. Kavukcuoglu, P. Kohli, and D. Hassabis, “Highly accurate protein structure prediction with alphafold,” Nature 596
2021
Later among the works it cites.
D. A. Roberts, S. Yaida, and B. Hanin, “The Principles of Deep Learning Theory,” (2021), 10.1017/9781009023405 , arXiv:2106.10165 [cs.LG]
Original
2021
Later among the works it cites.
J. Erdmenger, K. T. Grosvenor, and R. Jefferson, “Towards quantifying information flows: relative entropy in deep neural networks and the renormalization group,” SciPost Phys. 12

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

A. Géron, Hands-on Machine Learning with Scikit-Learn, Keras, and TensorFlow: Unsupervised learning techniques (O’Reilly Media, Incorporated, 2019)

2019

Cited alongside, same era.

J. Schrittwieser, I. Antonoglou, T. Hubert, K. Simonyan, L. Sifre, S. Schmitt, A. Guez, E. Lockhart, D. Hassabis, T. Graepel, T. Lillicrap, and D. Silver, “Mastering atari, go, chess and shogi by planning with a learned model,” Nature 588

2020

Cited alongside, same era.

Original

2022

Later among the works it cites.

K. T. Grosvenor and R. Jefferson, “The edge of chaos: quantum field theory and deep neural networks,” SciPost Phys. 12

Original

2022

Later among the works it cites.

R. Jefferson, “Criticality in deep neural nets,” https://rojefferson.blog/2020/06/19/criticality-in-deep-neural-nets/ (2020), accessed: 2022-11-17

2022

Later among the works it cites.