Fetching the paper…
Reading the bibliography…
Understanding the dynamics of neural network parameters during training is one of the key challenges in building a theoretical foundation for deep learning.
More is different
Philip W Anderson · 1972
Earlier work this paper cites.
The modified equation approach to the stability and accuracy analysis of finite-difference methods
RF Warming and BJ Hyett · 1974
Earlier work this paper cites.
Learning representations by back-propagating errors
David E Rumelhart, Geoffrey E Hinton, and Ronald J Williams · 1986
Earlier work this paper cites.
Neural networks and principal component analysis: Learning from examples without local minima
Pierre Baldi and Kurt Hornik · 1989
Earlier work this paper cites.
Dynamics of on-line gradient descent learning for multilayer neural networks
David Saad and Sara Solla · 1995
Earlier work this paper cites.
The role of symmetry in fundamental physics
David J Gross · 1996
Earlier work this paper cites.
Stochastic approximation and recursive algorithms and applications , volume 35
Harold Kushner and G George Yin · 2003
Earlier work this paper cites.
Exact solutions to the nonlinear dynamics of learning in deep linear neural networks
Andrew M Saxe, James L McClelland, and Surya Ganguli · 2013
Earlier work this paper cites.
A differential equation for modeling nesterov’s accelerated gradient method: Theory and insights
Weijie Su, Stephen Boyd, and Emmanuel Candes · 2014
Earlier work this paper cites.
On pixel-wise explanations for non-linear classifier decisions by layer-wise relevance propagation
Sebastian Bach, Alexander Binder, Grégoire Montavon, Frederick Klauschen, Klaus-Robert Müller, and Wojciech Samek · 2015
Earlier work this paper cites.
Batch normalization: Accelerating deep network training by reducing internal covariate shift
Sergey Ioffe and Christian Szegedy · 2015
Earlier work this paper cites.
Continuous-time limit of stochastic gradient descent revisited
Stephan Mandt, Matthew D Hoffman, and David M Blei · 2015
Earlier work this paper cites.
Path-sgd: Path-normalized optimization in deep neural networks
Behnam Neyshabur, Russ R Salakhutdinov, and Nati Srebro · 2015
Earlier work this paper cites.
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton · 2016
Earlier work this paper cites.
Deep learning without poor local minima
Kenji Kawaguchi · 2016
Earlier work this paper cites.
Weight normalization: A simple reparameterization to accelerate training of deep neural networks
Tim Salimans and Durk P Kingma · 2016
Earlier work this paper cites.
Riemannian approach to batch normalization
Minhyung Cho and Jaehyung Lee · 2017
Cited alongside, same era.
Three factors influencing minima in sgd
Stanisław Jastrzębski, Zachary Kenton, Devansh Arpit, Nicolas Ballas, Asja Fischer, Yoshua Bengio, and Amos Storkey · 2017
Cited alongside, same era.
Stochastic modified equations and adaptive stochastic gradient algorithms
Qianxiao Li, Cheng Tai, and E Weinan · 2017
Cited alongside, same era.
Stochastic gradient descent as approximate bayesian inference
Stephan Mandt, Matthew D Hoffman, and David M Blei · 2017
Cited alongside, same era.
A bayesian perspective on generalization and stochastic gradient descent
Samuel L Smith and Quoc V Le · 2017
Cited alongside, same era.
Online normalization for training neural networks
Vitaliy Chiley, Ilya Sharapov, Atli Kosson, Urs Koster, Ryan Reece, Sofia Samaniego de la Fuente, Vishal Subbiah, and Michael James · 2019
Later among the works it cites.
Width provably matters in optimization for deep linear neural networks
Simon S Du and Wei Hu · 2019
Later among the works it cites.
Yuanyuan Feng, Tingran Gao, Lei Li, Jian-Guo Liu, and Yulong Lu · 2019
Later among the works it cites.
Dynamics of stochastic gradient descent for two-layer neural networks in the teacher-student setup
Sebastian Goldt, Madhu Advani, Andrew M Saxe, Florent Krzakala, and Lenka Zdeborová · 2019
Later among the works it cites.
alphaXiv searches the wider corpus for related work and actual follow-ups.
alphaXiv is searching for related work…
Twan Van Laarhoven · 2017
Cited alongside, same era.
Stochastic modified equations for the asynchronous stochastic gradient descent
Jing An, Jianfeng Lu, and Lexing Ying · 2018
Cited alongside, same era.
Stochastic gradient descent performs variational inference, converges to limit cycles for deep networks
Pratik Chaudhari and Stefano Soatto · 2018
Cited alongside, same era.
Norm matters: efficient and accurate normalization schemes in deep networks
Elad Hoffer, Ron Banner, Itay Golan, and Daniel Soudry · 2018
Cited alongside, same era.
Neural tangent kernel: Convergence and generalization in neural networks
Arthur Jacot, Franck Gabriel, and Clément Hongler · 2018
Cited alongside, same era.
An analytic theory of generalization dynamics and transfer learning in deep linear networks
Andrew K Lampinen and Surya Ganguli · 2018
Cited alongside, same era.
A mean field view of the landscape of two-layer neural networks
Song Mei, Andrea Montanari, and Phan-Minh Nguyen · 2018
Cited alongside, same era.
Nikola B Kovachki and Andrew M Stuart · 2019
Later among the works it cites.
Loss landscapes of regularized linear autoencoders
Daniel Kunin, Jonathan M Bloom, Aleksandrina Goeva, and Cotton Seed · 2019
Later among the works it cites.
Wide neural networks of any depth evolve as linear models under gradient descent
Jaehoon Lee, Lechao Xiao, Samuel Schoenholz, Yasaman Bahri, Roman Novak, Jascha Sohl-Dickstein, and Jeffrey Pennington · 2019
Later among the works it cites.
An exponential learning rate schedule for deep learning
Zhiyuan Li and Sanjeev Arora · 2019
Later among the works it cites.
Fisher-rao metric, geometry, and complexity of neural networks
Tengyuan Liang, Tomaso Poggio, Alexander Rakhlin, and James Stokes · 2019
Later among the works it cites.
A mathematical theory of semantic development in deep neural networks
Andrew M Saxe, James L McClelland, and Surya Ganguli · 2019
Later among the works it cites.
Implicit gradient regularization
David GT Barrett and Benoit Dherin · 2020
Closest in time.
Deep learning versus kernel learning: an empirical study of loss landscape geometry and the time evolution of the neural tangent kernel
Stanislav Fort, Gintare Karolina Dziugaite, Mansheej Paul, Sepideh Kharaghani, Daniel M Roy, and Surya Ganguli · 2020
Closest in time.
Reconciling modern deep learning with traditional optimization analyses: The intrinsic learning rate
Zhiyuan Li, Kaifeng Lyu, and Sanjeev Arora · 2020
Closest in time.
Pruning neural networks without any data by iteratively conserving synaptic flow
Hidenori Tanaka, Daniel Kunin, Daniel LK Yamins, and Surya Ganguli · 2020
Closest in time.
Spherical motion dynamics of deep neural networks with batch normalization and weight decay
Ruosi Wan, Zhanxing Zhu, Xiangyu Zhang, and Jian Sun · 2020
Closest in time.
On the validity of modeling sgd with stochastic differential equations (sdes)
Zhiyuan Li, Sadhika Malladi, and Sanjeev Arora · 2021
Closest in time.