We study the conjectured relationship between the implicit regularization in neural networks, trained with gradient-based methods, and rank minimization of their weight matrices.
Reading the bibliography…
The effective rank: A measure of effective dimensionality
O. Roy and M. Vetterli · 2007
Earlier work this paper cites.
Sampling from large matrices: An approach through geometric functional analysis
M. Rudelson and R. Vershynin · 2007
Earlier work this paper cites.
Exploiting linear structure within convolutional networks for efficient evaluation
E. L. Denton, W. Zaremba, J. Bruna, Y. LeCun, and R. Fergus · 2014
Earlier work this paper cites.
In search of the real inductive bias: On the role of implicit regularization in deep learning
B. Neyshabur, R. Tomioka, and N. Srebro · 2015
Earlier work this paper cites.
Compression-aware training of deep networks
J. M. Alvarez and M. Salzmann · 2017
Earlier work this paper cites.
Exploring generalization in deep learning
B. Neyshabur, S. Bhojanapalli, D. McAllester, and N. Srebro · 2017
Earlier work this paper cites.
On compressing deep models by low rank and sparse decomposition
X. Yu, T. Liu, X. Wang, and D. Tao · 2017
Earlier work this paper cites.
Understanding deep learning requires rethinking generalization
C. Zhang, S. Bengio, M. Hardt, B. Recht, and O. Vinyals · 2017
Earlier work this paper cites.
Stronger generalization bounds for deep nets via a compression approach
S. Arora, R. Ge, B. Neyshabur, and Y. Zhang · 2018
Earlier work this paper cites.
Algorithmic regularization in learning deep homogeneous models: Layers are automatically balanced
S. S. Du, W. Hu, and J. D. Lee · 2018
Earlier work this paper cites.
Implicit regularization in matrix factorization
S. Gunasekar, B. Woodworth, S. Bhojanapalli, B. Neyshabur, and N. Srebro · 2018
Earlier work this paper cites.
Algorithmic regularization in over-parameterized matrix sensing and neural networks with quadratic activations
Y. Li, T. Ma, and H. Zhang · 2018
Earlier work this paper cites.
Implicit regularization in nonconvex statistical estimation: Gradient descent converges linearly for phase retrieval and matrix completion
C. Ma, K. Wang, Y. Chi, and Y. Chen · 2018
Earlier work this paper cites.
The implicit bias of gradient descent on separable data
D. Soudry, E. Hoffer, M. S. Nacson, S. Gunasekar, and N. Srebro · 2018
Cited alongside, same era.
Implicit regularization in deep matrix factorization
S. Arora, N. Cohen, W. Hu, and Y. Luo · 2019
Cited alongside, same era.
Implicit regularization of discrete gradient dynamics in linear neural networks
G. Gidel, F. Bach, and S. Lacoste-Julien · 2019
Cited alongside, same era.
Gradient descent maximizes the margin of homogeneous neural networks
K. Lyu and J. Li · 2019
Cited alongside, same era.
Overparameterized nonlinear learning: Gradient descent takes the shortest path?
S. Oymak and M. Soltanolkotabi · 2019
Cited alongside, same era.
Stable rank normalization for improved generalization in neural networks and gans
A. Sanyal, P. H. Torr, and P. K. Dokania · 2019
Cited alongside, same era.
Gradient dynamics of shallow univariate relu networks
F. Williams, M. Trager, D. Panozzo, C. Silva, D. Zorin, and J. Bruna · 2019
Cited alongside, same era.
On implicit regularization: Morse functions and applications to matrix factorization
M. A. Belabbas · 2020
Cited alongside, same era.
Implicit bias of gradient descent for wide two-layer neural networks trained with the logistic loss
L. Chizat and F. Bach · 2020
Cited alongside, same era.
Implicit regularization in matrix sensing: A geometric view leads to stronger results
A. Eftekhari and K. Zygalakis · 2020
Cited alongside, same era.
Directional convergence and alignment in deep learning
Z. Ji and M. Telgarsky · 2020
Cited alongside, same era.
Gradient descent follows the regularization path for general losses
Z. Ji, M. Dudík, R. E. Schapire, and M. Telgarsky · 2020
Cited alongside, same era.
Towards resolving the implicit bias of gradient descent for matrix factorization: Greedy low-rank learning
Z. Li, Y. Luo, and K. Lyu · 2020
Later among the works it cites.
Implicit bias in deep linear classification: Initialization scale vs training accuracy
E. Moroshko, B. E. Woodworth, S. Gunasekar, J. D. Lee, N. Srebro, and D. Soudry · 2020
Later among the works it cites.
Implicit regularization in deep learning may not be explainable by norms
N. Razin and N. Cohen · 2020
Later among the works it cites.
Compressed deep networks: Goodbye svd, hello robust low-rank approximation
M. Tukan, A. Maalouf, M. Weksler, and D. Feldman · 2020
Later among the works it cites.
Kernel and rich regimes in overparametrized models
B. Woodworth, S. Gunasekar, J. D. Lee, E. Moroshko, P. Savarese, I. Golan, D. Soudry, and N. Srebro · 2020
Later among the works it cites.
A unifying view on implicit bias in training linear neural networks
C. Yun, S. Krishnan, and H. Mobahi · 2020
Later among the works it cites.
On the implicit bias of initialization shape: Beyond infinitesimal mirror descent
S. Azulay, E. Moroshko, M. S. Nacson, B. Woodworth, N. Srebro, A. Globerson, and D. Soudry · 2021
Later among the works it cites.
Characterizing the implicit bias via a primal-dual analysis
Z. Ji and M. Telgarsky · 2021
Later among the works it cites.
Implicit regularization in tensor factorization
N. Razin, A. Maman, and N. Cohen · 2021
Later among the works it cites.
Gradient methods never overfit on separable data
O. Shamir · 2021
Later among the works it cites.
Implicit regularization in relu networks with the square loss
G. Vardi and O. Shamir · 2021
Later among the works it cites.
On margin maximization in linear and relu networks
G. Vardi, O. Shamir, and N. Srebro · 2021
Later among the works it cites.
alphaXiv searches the wider corpus for related work and actual follow-ups.
alphaXiv is searching for related work…