2022

Limitations of the NTK for Understanding Generalization in Deep Learning

Vyas, Nikhil, Bansal, Yamini, Nakkiran, Preetum

Understand

The ``Neural Tangent Kernel'' (NTK) (Jacot et al 2018), and its empirical variants have been proposed as a proxy to capture certain behaviors of real neural networks.

In this work, we study NTKs through the lens of scaling laws, and demonstrate that they fall short of explaining important aspects of neural network generalization.
In particular, we demonstrate realistic settings where finite-width neural networks have significantly better data scaling exponents as compared to their corresponding empirical and infinite NTKs at initialization.
This reveals a more fundamental difference between the real networks and NTKs, beyond just a few percentage points of test accuracy.

Built on

Reading digits in natural images with unsupervised feature learning
Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y Ng · 2011
Earlier work this paper cites.
Learning and generalization in overparameterized neural networks, going beyond two layers
Original
Zeyuan Allen-Zhu, Yuanzhi Li, and Yingyu Liang · 2018
Earlier work this paper cites.
Gradient descent provably optimizes over-parameterized neural networks
Original
Simon S Du, Xiyu Zhai, Barnabas Poczos, and Aarti Singh · 2018
Earlier work this paper cites.
Neural tangent kernel: Convergence and generalization in neural networks
Arthur Jacot, Clément Hongler, and Franck Gabriel · 2018
Earlier work this paper cites.
Fine-grained analysis of optimization and generalization for overparameterized two-layer neural networks
Sanjeev Arora, Simon Du, Wei Hu, Zhiyuan Li, and Ruosong Wang · 2019
Earlier work this paper cites.
On exact computation with an infinitely wide neural net
Sanjeev Arora, Simon S. Du, Wei Hu, Zhiyuan Li, Ruslan Salakhutdinov, and Ruosong Wang · 2019
Earlier work this paper cites.
A mean-field limit for certain deep neural networks, 2019
Dyego Araújo, Roberto I. Oliveira, and Daniel Yukimura · 2019
Earlier work this paper cites.
Reconciling modern machine learning practice and the bias-variance trade-off, 2019
Mikhail Belkin, Daniel Hsu, Siyuan Ma, and Soumik Mandal · 2019
Earlier work this paper cites.
Gradient descent finds global minima of deep neural networks
Simon Du, Jason Lee, Haochuan Li, Liwei Wang, and Xiyu Zhai · 2019
Earlier work this paper cites.
Jamming transition as a paradigm to understand the loss landscape of deep neural networks
Mario Geiger, Stefano Spigler, Stéphane d’ Ascoli, Levent Sagun, Marco Baity-Jesi, Giulio Biroli, and Matthieu Wyart · 2019
Earlier work this paper cites.
Wide neural networks of any depth evolve as linear models under gradient descent
Jaehoon Lee, Lechao Xiao, Samuel S. Schoenholz, Yasaman Bahri, Roman Novak, Jascha Sohl-Dickstein, and Jeffrey Pennington · 2019
Earlier work this paper cites.
A constructive prediction of the generalization error across scales
Original
Jonathan S Rosenfeld, Amir Rosenfeld, Yonatan Belinkov, and Nir Shavit · 2019
Earlier work this paper cites.
Mean field analysis of deep neural networks, 2019
Justin Sirignano and Konstantinos Spiliopoulos · 2019
Earlier work this paper cites.
Asymptotics of wide convolutional neural networks
Original
Anders Andreassen and Ethan Dyer · 2020
Earlier work this paper cites.
The neural tangent kernel in high dimensions: Triple descent and a multi-scale theory of generalization
Ben Adlam and Jeffrey Pennington · 2020
Earlier work this paper cites.
Spectrum dependent learning curves in kernel regression and wide neural networks
Blake Bordelon, Abdulkadir Canatar, and Cengiz Pehlevan · 2020
Earlier work this paper cites.

Similar

Asymptotics of wide networks from feynman diagrams
Ethan Dyer and Guy Gur-Ari · 2020
Cited alongside, same era.
Learning parities with neural networks
Amit Daniely and Eran Malach · 2020
Cited alongside, same era.
Triple descent and the two kinds of overfitting: where & why do they appear?
Stéphane d’Ascoli, Levent Sagun, and Giulio Biroli · 2020
Cited alongside, same era.
Deep learning versus kernel learning: an empirical study of loss landscape geometry and the time evolution of the neural tangent kernel
Stanislav Fort, Gintare Karolina Dziugaite, Mansheej Paul, Sepideh Kharaghani, Daniel M. Roy, and Surya Ganguli · 2020
Cited alongside, same era.
Modeling from features: a mean-field framework for over-parameterized deep neural networks, 2020
Cong Fang, Jason D. Lee, Pengkun Yang, and Tong Zhang · 2020
Cited alongside, same era.
When do neural networks outperform kernel methods?
Behrooz Ghorbani, Song Mei, Theodor Misiakiewicz, and Andrea Montanari · 2020
Cited alongside, same era.
Scaling laws for neural language models
Original
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei · 2020
Cited alongside, same era.
Finite versus infinite neural networks: an empirical study
Jaehoon Lee, Samuel S. Schoenholz, Jeffrey Pennington, Ben Adlam, Lechao Xiao, Roman Novak, and Jascha Sohl-Dickstein · 2020
Cited alongside, same era.
Deep double descent: Where bigger models and more data hurt
Preetum Nakkiran, Gal Kaplun, Yamini Bansal, Tristan Yang, Boaz Barak, and Ilya Sutskever · 2020
Cited alongside, same era.
A rigorous framework for the mean field limit of multilayer neural networks
Original
Phan-Minh Nguyen and Huy Tuan Pham · 2020
Cited alongside, same era.
Neural tangents: Fast and easy infinite neural networks in python
Roman Novak, Lechao Xiao, Jiri Hron, Jaehoon Lee, Alexander A. Alemi, Jascha Sohl-Dickstein, and Samuel S. Schoenholz · 2020
Cited alongside, same era.
Neural kernels without tangents
Vaishaal Shankar, Alex Fang, Wenshuo Guo, Sara Fridovich-Keil, Jonathan Ragan-Kelley, Ludwig Schmidt, and Benjamin Recht · 2020
Cited alongside, same era.

Then

Explaining neural scaling laws
Original
Yasaman Bahri, Ethan Dyer, Jared Kaplan, Jaehoon Lee, and Utkarsh Sharma · 2021
Later among the works it cites.
Spectral bias and task-model alignment explain generalization in kernel regression and infinitely wide neural networks
Abdulkadir Canatar, Blake Bordelon, and Cengiz Pehlevan · 2021
Later among the works it cites.
Local signal adaptivity: Provable feature learning in neural networks beyond kernels
Stefani Karp, Ezra Winston, Yuanzhi Li, and Aarti Singh · 2021
Later among the works it cites.
Properties of the after kernel
Original
Philip M. Long · 2021
Later among the works it cites.
Provable convergence of nesterov accelerated method for over-parameterized neural networks
Original
Xin Liu and Zhisong Pan · 2021
Later among the works it cites.
Towards an understanding of benign overfitting in neural networks
Original
Zhu Li, Zhi-Hua Zhou, and Arthur Gretton · 2021
Later among the works it cites.
The deep bootstrap framework: Good online learners are good offline generalizers
Preetum Nakkiran, Behnam Neyshabur, and Hanie Sedghi · 2021
Later among the works it cites.
What can linearized neural networks actually say about generalization?
Guillermo Ortiz-Jiménez, Seyed-Mohsen Moosavi-Dezfooli, and Pascal Frossard · 2021
Later among the works it cites.
Geometric compression of invariant manifolds in neural networks
Jonas Paccolat, Leonardo Petrini, Mario Geiger, Kevin Tyloo, and Matthieu Wyart · 2021
Later among the works it cites.
The principles of deep learning theory
Original
Daniel A. Roberts, Sho Yaida, and Boris Hanin · 2021
Later among the works it cites.
Neural tangent kernel eigenvalues accurately predict generalization
Original
James B. Simon, Madeline Dickens, and Michael Robert DeWeese · 2021
Later among the works it cites.
Explicit loss asymptotics in the gradient descent training of neural networks
Maksim Velikanov and Dmitry Yarotsky · 2021
Later among the works it cites.
Tensor programs IV: feature learning in infinite-width neural networks
Greg Yang and Edward J. Hu · 2021
Later among the works it cites.
Data scaling laws in NMT: the effect of noise and architecture
Original
Yamini Bansal, Behrooz Ghorbani, Ankush Garg, Biao Zhang, Maxim Krikun, Colin Cherry, Behnam Neyshabur, and Orhan Firat · 2022
Closest in time.
Learning curves for SGD on structured features
Blake Bordelon and Cengiz Pehlevan · 2022
Closest in time.
How to train your resnet 4: Architecture
David C. Page · 2022
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…