Understand
In this paper, we revisit implicit regularization from the ground up using notions from dynamical systems and invariant subspaces of Morse functions.
- The key contributions are a new criterion for implicit regularization---a leading contender to explain the generalization power of deep models such as neural networks---and a general blueprint to study it.
- We apply these techniques to settle a conjecture on implicit regularization in matrix factorization.
Built on
Ensembles semi-analytiques
S. Lojasiewicz · 1965
Earlier work this paper cites.
Dynamical systems that sort lists, diagonalize matrices, and solve linear programming problems
Roger W Brockett · 1991
Earlier work this paper cites.
The implicit bias of gradient descent on separable data
Daniel Soudry, Elad Hoffer, Mor Shpigel Nacson, Suriya Gunasekar, and Nathan Srebro · 2001
Earlier work this paper cites.
Optimization and dynamical systems
Uwe Helmke and John B Moore · 2012
Earlier work this paper cites.
Similar
Lectures on Morse homology
Augustin Banyaga and David Hurtubise · 2013
Cited alongside, same era.
In search of the real inductive bias: On the role of implicit regularization in deep learning
Behnam Neyshabur, Ryota Tomioka, and Nathan Srebro · 2014
Cited alongside, same era.
Morse theory.(AM-51)
John Milnor · 2016
Cited alongside, same era.
Then
Implicit regularization in matrix factorization
Suriya Gunasekar, Blake E Woodworth, Srinadh Bhojanapalli, Behnam Neyshabur, and Nati Srebro · 2017
Later among the works it cites.
Geometry of optimization and implicit regularization in deep learning
Behnam Neyshabur, Ryota Tomioka, Ruslan Salakhutdinov, and Nathan Srebro · 2017
Later among the works it cites.
Implicit regularization in deep matrix factorization
Sanjeev Arora, Nadav Cohen, Wei Hu, and Yuping Luo · 2019
Later among the works it cites.
Beyond the bibliography
alphaXiv searches the wider corpus for related work and actual follow-ups.
alphaXiv is searching for related work…