2019

Harnessing the Power of Infinitely Wide Deep Nets on Small-data Tasks

Arora, Sanjeev, Du, Simon S., Li, Zhiyuan et al.

Understand

Recent research shows that the following two models are equivalent: (a) infinitely wide neural networks (NNs) trained under l2 loss by gradient descent with infinitesimally small learning rate (b) kernel regression with respect to so-called Neural Tangent Kernels (NTKs) (Jacot et al., 2018).

An efficient algorithm to compute the NTK, as well as its convolutional counterparts, appears in Arora et al.
(2019a), which allowed studying performance of infinitely wide nets on datasets like CIFAR-10.
However, super-quadratic running time of kernel methods makes them best suited for small-data tasks.

Built on

Fine-grained analysis of optimization and generalization for overparameterized two-layer neural networks
Original
Sanjeev Arora, Simon S Du, Wei Hu, Zhiyuan Li, and Ruosong Wang · 1901
Earlier work this paper cites.
A generalization theory of gradient descent for learning over-parameterized deep relu networks
Original
Yuan Cao and Quanquan Gu · 1902
Earlier work this paper cites.
On exact computation with an infinitely wide neural net
Original
Sanjeev Arora, Simon S Du, Wei Hu, Zhiyuan Li, Ruslan Salakhutdinov, and Ruosong Wang · 1904
Earlier work this paper cites.
Generalization bounds of stochastic gradient descent for wide and deep neural networks
Original
Yuan Cao and Quanquan Gu · 1905
Earlier work this paper cites.
Priors for infinite networks
Radford M Neal · 1996
Earlier work this paper cites.
Computing with infinite networks
Christopher KI Williams · 1997
Earlier work this paper cites.
Learning with kernels: support vector machines, regularization, optimization, and beyond
Bernhard Scholkopf and Alexander J Smola · 2001
Earlier work this paper cites.
Continuous neural networks
Nicolas Le Roux and Yoshua Bengio · 2007
Earlier work this paper cites.
Kernel methods for deep learning
Youngmin Cho and Lawrence K Saul · 2009
Earlier work this paper cites.
Imagenet: A large-scale hierarchical image database
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei · 2009
Earlier work this paper cites.
Do we need hundreds of classifiers to solve real world classification problems?
Manuel Fernández-Delgado, Eva Cernadas, Senén Barro, and Dinani Amorim · 2014
Earlier work this paper cites.

Similar

Stochastic gradient descent optimizes over-parameterized deep ReLU networks
Original
Difan Zou, Yuan Cao, Dongruo Zhou, and Quanquan Gu · 2014
Cited alongside, same era.
Fast and accurate deep network learning by exponential linear units (elus)
Original
Djork-Arné Clevert, Thomas Unterthiner, and Sepp Hochreiter · 2015
Cited alongside, same era.
Steps toward deep kernel methods from infinite neural networks
Original
Tamir Hazan and Tommi Jaakkola · 2015
Cited alongside, same era.
Toward deeper understanding of neural networks: The power of initialization and a dual view on expressivity
Amit Daniely, Roy Frostig, and Yoram Singer · 2016
Cited alongside, same era.
Deep residual learning for image recognition
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun · 2016
Cited alongside, same era.
Understanding deep learning requires rethinking generalization
Original
Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals · 2016
Cited alongside, same era.
SGD learns the conjugate kernel class of the network
Original
Amit Daniely · 2017
Cited alongside, same era.
Split-brain autoencoders: Unsupervised learning by cross-channel prediction
Richard Zhang, Phillip Isola, and Alexei A Efros · 2017
Cited alongside, same era.
Neural tangent kernel: Convergence and generalization in neural networks
Arthur Jacot, Franck Gabriel, and Clément Hongler · 2018
Cited alongside, same era.
Deep neural networks as gaussian processes
Jaehoon Lee, Jascha Sohl-dickstein, Jeffrey Pennington, Roman Novak, Sam Schoenholz, and Yasaman Bahri · 2018
Cited alongside, same era.
Learning overparameterized neural networks via stochastic gradient descent on structured data
Original
Yuanzhi Li and Yingyu Liang · 2018
Cited alongside, same era.
Learning and generalization in overparameterized neural networks, going beyond two layers
Original
Zeyuan Allen-Zhu, Yuanzhi Li, and Yingyu Liang
Cited in the paper.

Then

Gaussian process behaviour in wide deep neural networks
Original
Alexander G de G Matthews, Mark Rowland, Jiri Hron, Richard E Turner, and Zoubin Ghahramani · 2018
Later among the works it cites.
Modern neural networks generalize on small data sets
Matthew Olson, Abraham Wyner, and Richard Berk · 2018
Later among the works it cites.
On the inductive bias of neural tangent kernels
Original
Alberto Bietti and Julien Mairal · 2019
Closest in time.
Graph neural tangent kernel: Fusing graph neural networks with graph kernels
Original
Simon S. Du, Kangcheng Hou, Barnabás Póczos, Ruslan Salakhutdinov, Ruosong Wang, and Keyulu Xu · 2019
Closest in time.
Deep convolutional networks as shallow gaussian processes
Adrià Garriga-Alonso, Carl Edward Rasmussen, and Laurence Aitchison · 2019
Closest in time.
Scaling and benchmarking self-supervised visual representation learning
Original
Priya Goyal, Dhruv Mahajan, Abhinav Gupta, and Ishan Misra · 2019
Closest in time.
Wide neural networks of any depth evolve as linear models under gradient descent
Original
Jaehoon Lee, Lechao Xiao, Samuel S Schoenholz, Yasaman Bahri, Jascha Sohl-Dickstein, and Jeffrey Pennington · 2019
Closest in time.
Bayesian deep convolutional networks with many channels are gaussian processes
Roman Novak, Lechao Xiao, Yasaman Bahri, Jaehoon Lee, Greg Yang, Daniel A. Abolafia, Jeffrey Pennington, and Jascha Sohl-dickstein · 2019
Closest in time.
The effect of network width on stochastic gradient descent and generalization: an empirical study
Original
Daniel S Park, Jascha Sohl-Dickstein, Quoc V Le, and Samuel L Smith · 2019
Closest in time.
Scaling limits of wide neural networks with weight sharing: Gaussian process behavior, gradient independence, and neural tangent kernel derivation
Original
Greg Yang · 2019
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…