Built on

Implicit Regularization in Deep Matrix Factorization
Original
Sanjeev Arora, Nadav Cohen, Wei Hu, and Yuping Luo · 1905
Earlier work this paper cites.
The Implicit Bias of Depth: How Incremental Learning Drives Generalization, December 2019
Original
Daniel Gissin, Shai Shalev-Shwartz, and Amit Daniely · 1909
Earlier work this paper cites.
Drawing Early-Bird Tickets: Towards More Efficient Training of Deep Networks, February 2022
Original
Haoran You, Chaojian Li, Pengfei Xu, Yonggan Fu, Yue Wang, Xiaohan Chen, Richard G. Baraniuk, Zhangyang Wang, and Yingyan Lin · 1909
Earlier work this paper cites.
Towards Resolving the Implicit Bias of Gradient Descent for Matrix Factorization: Greedy Low-Rank Learning
Original
Zhiyuan Li, Yuping Luo, and Kaifeng Lyu · 2012
Earlier work this paper cites.
Exact solutions to the nonlinear dynamics of learning in deep linear neural networks
Original
Andrew M. Saxe, James L. McClelland, and Surya Ganguli · 2014
Earlier work this paper cites.
Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun · 2015
Earlier work this paper cites.
Training CNNs with Low-Rank Filters for Efficient Image Classification, February 2016
Original
Yani Ioannou, Duncan Robertson, Jamie Shotton, Roberto Cipolla, and Antonio Criminisi · 2016
Earlier work this paper cites.
Characterizing Implicit Bias in Terms of Optimization Geometry
Suriya Gunasekar, Jason Lee, Daniel Soudry, and Nathan Srebro · 2018
Earlier work this paper cites.

Similar

Measuring the Intrinsic Dimension of Objective Landscapes
Original
Chunyuan Li, Heerad Farkhoor, Rosanne Liu, and Jason Yosinski · 2018
Cited alongside, same era.
On the Spectral Bias of Neural Networks
Original
Nasim Rahaman, Aristide Baratin, Devansh Arpit, Felix Draxler, Min Lin, Fred A. Hamprecht, Yoshua Bengio, and Aaron Courville · 2019
Cited alongside, same era.
PowerSGD: Practical Low-Rank Gradient Compression for Distributed Optimization
Thijs Vogels, Sai Praneeth Karimireddy, and Martin Jaggi · 2019
Cited alongside, same era.
Learning Low-rank Deep Neural Networks via Singular Vector Orthogonality Regularization and Singular Value Sparsification
Huanrui Yang, Minxue Tang, Wei Wen, Feng Yan, Daniel Hu, Ang Li, Hai Li, and Yiran Chen · 2020
Cited alongside, same era.
Low-Rank Compression of Neural Nets: Learning the Rank of Each Layer
Yerlan Idelbayev and Miguel A. Carreira-Perpinan · 2020

Then

PINNup: Robust neural network wavefield solutions using frequency upscaling and neuron splitting
Original
Xinquan Huang and Tariq Alkhalifah · 2021
Later among the works it cites.
ZerO Initialization: Initializing Residual Networks with only Zeros and Ones, October 2021
Original
Jiawei Zhao, Florian Schäfer, and Anima Anandkumar · 2021
Later among the works it cites.
Saddle-to-Saddle Dynamics in Deep Linear Networks: Small Initialization Training, Symmetry, and Sparsity
Original
Arthur Jacot, François Ged, Berfin Şimşek, Clément Hongler, and Franck Gabriel · 2022
Later among the works it cites.
Monarch: Expressive Structured Matrices for Efficient and Accurate Training, April 2022
Original
Tri Dao, Beidi Chen, Nimit Sohoni, Arjun Desai, Michael Poli, Jessica Grogan, Alexander Liu, Aniruddh Rao, Atri Rudra, and Christopher Ré · 2022
Later among the works it cites.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…

2023

InRank: Incremental Low-Rank Learning

Zhao, Jiawei, Zhang, Yifei, Chen, Beidi et al.

Understand

The theory of greedy low-rank learning (GLRL) aims to explain the impressive generalization capabilities of deep learning.

It proves that stochastic gradient-based training implicitly regularizes neural networks towards low-rank solutions through a gradual increase of the rank during training.
However, there is a gap between theory and practice since GLRL requires an infinitesimal initialization of the weights, which is not practical due to the fact that it is a saddle point.
In this work, we remove the assumption of infinitesimal initialization by focusing on cumulative weight updates.

Built on

Implicit Regularization in Deep Matrix Factorization
Original
Sanjeev Arora, Nadav Cohen, Wei Hu, and Yuping Luo · 1905
Earlier work this paper cites.
The Implicit Bias of Depth: How Incremental Learning Drives Generalization, December 2019
Original
Daniel Gissin, Shai Shalev-Shwartz, and Amit Daniely · 1909
Earlier work this paper cites.
Drawing Early-Bird Tickets: Towards More Efficient Training of Deep Networks, February 2022
Original
Haoran You, Chaojian Li, Pengfei Xu, Yonggan Fu, Yue Wang, Xiaohan Chen, Richard G. Baraniuk, Zhangyang Wang, and Yingyan Lin · 1909
Earlier work this paper cites.
Towards Resolving the Implicit Bias of Gradient Descent for Matrix Factorization: Greedy Low-Rank Learning
Original
Zhiyuan Li, Yuping Luo, and Kaifeng Lyu · 2012
Earlier work this paper cites.
Exact solutions to the nonlinear dynamics of learning in deep linear neural networks
Original
Andrew M. Saxe, James L. McClelland, and Surya Ganguli · 2014
Earlier work this paper cites.
Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun · 2015
Earlier work this paper cites.
Training CNNs with Low-Rank Filters for Efficient Image Classification, February 2016
Original
Yani Ioannou, Duncan Robertson, Jamie Shotton, Roberto Cipolla, and Antonio Criminisi · 2016
Earlier work this paper cites.
Characterizing Implicit Bias in Terms of Optimization Geometry
Suriya Gunasekar, Jason Lee, Daniel Soudry, and Nathan Srebro · 2018
Earlier work this paper cites.

Similar

Measuring the Intrinsic Dimension of Objective Landscapes
Original
Chunyuan Li, Heerad Farkhoor, Rosanne Liu, and Jason Yosinski · 2018
Cited alongside, same era.
On the Spectral Bias of Neural Networks
Original
Nasim Rahaman, Aristide Baratin, Devansh Arpit, Felix Draxler, Min Lin, Fred A. Hamprecht, Yoshua Bengio, and Aaron Courville · 2019
Cited alongside, same era.
PowerSGD: Practical Low-Rank Gradient Compression for Distributed Optimization
Thijs Vogels, Sai Praneeth Karimireddy, and Martin Jaggi · 2019
Cited alongside, same era.
Learning Low-rank Deep Neural Networks via Singular Vector Orthogonality Regularization and Singular Value Sparsification
Huanrui Yang, Minxue Tang, Wei Wen, Feng Yan, Daniel Hu, Ang Li, Hai Li, and Yiran Chen · 2020
Cited alongside, same era.
Low-Rank Compression of Neural Nets: Learning the Rank of Each Layer
Yerlan Idelbayev and Miguel A. Carreira-Perpinan · 2020