Neural Networks Learning and Memorization with (almost) no Over-Parameterization

Computation with infinite neural networks

C.K.I. Williams · 1997

Earlier work this paper cites.

Weighted sums of random kitchen sinks: Replacing minimization with randomization in learning

A. Rahimi and B. Recht · 2009

Earlier work this paper cites.

Understanding the difficulty of training deep feedforward neural networks

X. Glorot and Y. Bengio · 2010

Earlier work this paper cites.

Introduction to the non-asymptotic analysis of random matrices

Original

Roman Vershynin · 2010

Earlier work this paper cites.

Learning polynomials with neural networks

A. Andoni, R. Panigrahy, G. Valiant, and L. Zhang · 2014

Earlier work this paper cites.

Understanding Machine Learning: From Theory to Algorithms

S. Shalev-Shwartz and S. Ben-David · 2014

Earlier work this paper cites.

Toward deeper understanding of neural networks: The power of initialization and a dual view on expressivity

Amit Daniely, Roy Frostig, and Yoram Singer · 2016

Earlier work this paper cites.

Sgd learns the conjugate class of the netwirk

Amit Daniely · 2017

Cited alongside, same era.

Sgd learns over-parameterized networks that provably generalize on linearly separable data

Alon Brutzkus, Amir Globerson, Eran Malach, and Shai Shalev-Shwartz · 2018

Cited alongside, same era.

Gradient descent finds global minima of deep neural networks

Original

Simon S Du, Jason D Lee, Haochuan Li, Liwei Wang, and Xiyu Zhai · 2018

Cited alongside, same era.

Neural tangent kernel: Convergence and generalization in neural networks

Arthur Jacot, Franck Gabriel, and Clément Hongler · 2018

Cited alongside, same era.

Fine-grained analysis of optimization and generalization for overparameterized two-layer neural networks

Original

Sanjeev Arora, Simon S Du, Wei Hu, Zhiyuan Li, and Ruosong Wang · 2019

Cited alongside, same era.

Mildly overparametrized neural nets can memorize training data efficiently

Original

Rong Ge, Runzhe Wang, and Haoyu Zhao · 2019

Closest in time.

Polylogarithmic width suffices for gradient descent to achieve arbitrarily small test error with shallow relu networks

Original

Ziwei Ji and Matus Telgarsky · 2019

Closest in time.

A comparative analysis of the optimization and generalization property of two-layer neural network and random feature models under gradient descent dynamics

Original

Chao Ma, Lei Wu, et al · 2019

Closest in time.

Towards moderate overparameterization: global convergence guarantees for training shallow neural networks

Original

Samet Oymak and Mahdi Soltanolkotabi · 2019

Closest in time.

Quadratic suffices for over-parametrization via matrix chernoff bound