Built on

On the inductive bias of neural tangent kernels
Original
Bietti, A. and Mairal, J. (2019) · 1905
Earlier work this paper cites.
Quadratic suffices for over-parametrization via matrix chernoff bound
Original
Song, Z. and Yang, X. (2019) · 1906
Earlier work this paper cites.
Dynamics of deep neural networks and neural tangent hierarchy
Original
Huang, J. and Yau, H.-T. (2019) · 1909
Earlier work this paper cites.
Harnessing the power of infinitely wide deep nets on small-data tasks
Original
Arora, S., Du, S. S., Li, Z., Salakhutdinov, R., Wang, R., and Yu, D. (2019c) · 1910
Earlier work this paper cites.
Tensor programs i: Wide feedforward or recurrent neural networks of any architecture are gaussian processes
Original
Yang, G. (2019) · 1910
Earlier work this paper cites.
Neural tangents: Fast and easy infinite neural networks in python
Original
Novak, R., Xiao, L., Hron, J., Lee, J., Alemi, A. A., Sohl-Dickstein, J., and Schoenholz, S. S. (2019) · 1912
Earlier work this paper cites.
A correspondence between bayesian estimation on stochastic processes and smoothing by splines
Kimeldorf, G. S. and Wahba, G. (1970) · 1970
Earlier work this paper cites.
On the local minima free condition of backpropagation learning
Yu, X.-H. and Chen, G.-A. (1995) · 1995
Earlier work this paper cites.
Nas-bench-201: Extending the scope of reproducible neural architecture search
Original
Dong, X. and Yang, Y. (2020) · 2001
Earlier work this paper cites.
Dynamically stable infinite-width limits of neural classifiers
Original
Golikov, E. A. (2020a) · 2006
Earlier work this paper cites.
Kernel methods through the roof: handling billions of points efficiently
Original
Meanti, G., Carratino, L., Rosasco, L., and Rudi, A. (2020) · 2006
Earlier work this paper cites.
On the almost sure convergence of stochastic gradient descent in non-convex problems
Original
Mertikopoulos, P., Hallak, N., Kavis, A., and Cevher, V. (2020) · 2006
Earlier work this paper cites.
Fourier features let networks learn high frequency functions in low dimensional domains
Original
Tancik, M., Srinivasan, P. P., Mildenhall, B., Fridovich-Keil, S., Raghavan, N., Singhal, U., Ramamoorthi, R., Barron, J. T., and Ng, R. (2020) · 2006
Earlier work this paper cites.
Tensor programs ii: Neural tangent kernel for any architecture
Original
Yang, G. (2020a) · 2006
Earlier work this paper cites.
On the similarity between the laplace and neural tangent kernels
Original
Geifman, A., Yadav, A., Kasten, Y., Galun, M., Jacobs, D., and Basri, R. (2020) · 2007
Earlier work this paper cites.

Similar

Deep neural tangent kernel and laplace kernel have the same rkhs
Original
Chen, L. and Xu, S. (2020) · 2009
Cited alongside, same era.
Tensor programs iii: Neural matrix laws
Original
Yang, G. (2020b) · 2009
Cited alongside, same era.
Label-aware neural tangent kernel: Toward better generalization and local elasticity
Original
Chen, S., He, H., and Su, W. J. (2020) · 2010
Cited alongside, same era.
Deep learning versus kernel learning: an empirical study of loss landscape geometry and the time evolution of the neural tangent kernel
Original
Fort, S., Dziugaite, G. K., Paul, M., Kharaghani, S., Roy, D. M., and Ganguli, S. (2020) · 2010
Cited alongside, same era.
Towards nngp-guided neural architecture search

Then

A convergence theory for deep learning via over-parameterization
Allen-Zhu, Z., Li, Y., and Song, Z. (2019) · 2019
Later among the works it cites.
Wide neural networks of any depth evolve as linear models under gradient descent
Lee, J., Xiao, L., Schoenholz, S., Bahri, Y., Novak, R., Sohl-Dickstein, J., and Pennington, J. (2019) · 2019
Later among the works it cites.
On connected sublevel sets in deep learning
Nguyen, Q. (2019) · 2019
Later among the works it cites.
Asymptotics of wide networks from feynman diagrams
Dyer, E. and Gur-Ari, G. (2020) · 2020
Later among the works it cites.
Infinite attention: Nngp and ntk for deep attention networks
Hron, J., Bahri, Y., Sohl-Dickstein, J., and Novak, R. (2020) · 2020
Later among the works it cites.
Neural kernels without tangents
Shankar, V., Fang, A., Guo, W., Fridovich-Keil, S., Ragan-Kelley, J., Schmidt, L., and Recht, B. (2020) · 2020

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…

2022

Neural Tangent Kernel: A Survey

Golikov, Eugene, Pokonechnyy, Eduard, Korviakov, Vladimir

Understand

A seminal work [Jacot et al., 2018] demonstrated that training a neural network under specific parameterization is equivalent to performing a particular kernel method as width goes to infinity.

This equivalence opened a promising direction for applying the results of the rich literature on kernel methods to neural nets which were much harder to tackle.
The present survey covers key results on kernel convergence as width goes to infinity, finite-width corrections, applications, and a discussion of the limitations of the corresponding method.

Built on

On the inductive bias of neural tangent kernels
Original
Bietti, A. and Mairal, J. (2019) · 1905
Earlier work this paper cites.
Quadratic suffices for over-parametrization via matrix chernoff bound
Original
Song, Z. and Yang, X. (2019) · 1906
Earlier work this paper cites.
Dynamics of deep neural networks and neural tangent hierarchy
Original
Huang, J. and Yau, H.-T. (2019) · 1909
Earlier work this paper cites.
Harnessing the power of infinitely wide deep nets on small-data tasks
Original
Arora, S., Du, S. S., Li, Z., Salakhutdinov, R., Wang, R., and Yu, D. (2019c) · 1910
Earlier work this paper cites.
Tensor programs i: Wide feedforward or recurrent neural networks of any architecture are gaussian processes
Original
Yang, G. (2019) · 1910
Earlier work this paper cites.
Neural tangents: Fast and easy infinite neural networks in python
Original
Novak, R., Xiao, L., Hron, J., Lee, J., Alemi, A. A., Sohl-Dickstein, J., and Schoenholz, S. S. (2019) · 1912
Earlier work this paper cites.
A correspondence between bayesian estimation on stochastic processes and smoothing by splines
Kimeldorf, G. S. and Wahba, G. (1970) · 1970
Earlier work this paper cites.
On the local minima free condition of backpropagation learning
Yu, X.-H. and Chen, G.-A. (1995) · 1995
Earlier work this paper cites.
Nas-bench-201: Extending the scope of reproducible neural architecture search
Original
Dong, X. and Yang, Y. (2020) · 2001
Earlier work this paper cites.
Dynamically stable infinite-width limits of neural classifiers
Original
Golikov, E. A. (2020a) · 2006
Earlier work this paper cites.
Kernel methods through the roof: handling billions of points efficiently
Original
Meanti, G., Carratino, L., Rosasco, L., and Rudi, A. (2020) · 2006
Earlier work this paper cites.
On the almost sure convergence of stochastic gradient descent in non-convex problems
Original
Mertikopoulos, P., Hallak, N., Kavis, A., and Cevher, V. (2020) · 2006
Earlier work this paper cites.
Fourier features let networks learn high frequency functions in low dimensional domains
Original
Tancik, M., Srinivasan, P. P., Mildenhall, B., Fridovich-Keil, S., Raghavan, N., Singhal, U., Ramamoorthi, R., Barron, J. T., and Ng, R. (2020) · 2006
Earlier work this paper cites.
Tensor programs ii: Neural tangent kernel for any architecture
Original
Yang, G. (2020a) · 2006
Earlier work this paper cites.
On the similarity between the laplace and neural tangent kernels
Original
Geifman, A., Yadav, A., Kasten, Y., Galun, M., Jacobs, D., and Basri, R. (2020) · 2007
Earlier work this paper cites.

Similar

Deep neural tangent kernel and laplace kernel have the same rkhs
Original
Chen, L. and Xu, S. (2020) · 2009
Cited alongside, same era.
Tensor programs iii: Neural matrix laws
Original
Yang, G. (2020b) · 2009
Cited alongside, same era.
Label-aware neural tangent kernel: Toward better generalization and local elasticity
Original
Chen, S., He, H., and Su, W. J. (2020) · 2010
Cited alongside, same era.
Deep learning versus kernel learning: an empirical study of loss landscape geometry and the time evolution of the neural tangent kernel
Original
Fort, S., Dziugaite, G. K., Paul, M., Kharaghani, S., Roy, D. M., and Ganguli, S. (2020) · 2010
Cited alongside, same era.
Towards nngp-guided neural architecture search

Then

A convergence theory for deep learning via over-parameterization
Allen-Zhu, Z., Li, Y., and Song, Z. (2019) · 2019
Later among the works it cites.
Wide neural networks of any depth evolve as linear models under gradient descent
Lee, J., Xiao, L., Schoenholz, S., Bahri, Y., Novak, R., Sohl-Dickstein, J., and Pennington, J. (2019) · 2019
Later among the works it cites.
On connected sublevel sets in deep learning
Nguyen, Q. (2019) · 2019
Later among the works it cites.
Asymptotics of wide networks from feynman diagrams
Dyer, E. and Gur-Ari, G. (2020) · 2020
Later among the works it cites.
Infinite attention: Nngp and ntk for deep attention networks
Hron, J., Bahri, Y., Sohl-Dickstein, J., and Novak, R. (2020) · 2020
Later among the works it cites.
Neural kernels without tangents
Shankar, V., Fang, A., Guo, W., Fridovich-Keil, S., Ragan-Kelley, J., Schmidt, L., and Recht, B. (2020) · 2020

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Original

Park, D. S., Lee, J., Peng, D., Cao, Y., and Sohl-Dickstein, J. (2020) · 2011

Cited alongside, same era.

Notes on deep learning theory

Original

Golikov, E. A. (2020b) · 2012

Cited alongside, same era.

Delving deep into rectifiers: Surpassing human-level performance on imagenet classification

He, K., Zhang, X., Ren, S., and Sun, J. (2015) · 2015

Cited alongside, same era.

Deep residual learning for image recognition

He, K., Zhang, X., Ren, S., and Sun, J. (2016) · 2016

Cited alongside, same era.

Gradient descent only converges to minimizers

Lee, J. D., Simchowitz, M., Jordan, M. I., and Recht, B. (2016) · 2016

Cited alongside, same era.

The loss surface of deep and wide neural networks

Nguyen, Q. and Hein, M. (2017) · 2017

Cited alongside, same era.

Gradient descent only converges to minimizers: Non-isolated critical points and invariant regions

Panageas, I. and Piliouras, G. (2017) · 2017

Cited alongside, same era.

JAX: composable transformations of Python+NumPy programs

Bradbury, J., Frostig, R., Hawkins, P., Johnson, M. J., Leary, C., Maclaurin, D., Necula, G., Paszke, A., VanderPlas, J., Wanderman-Milne, S., and Zhang, Q. (2018) · 2018

Cited alongside, same era.

Later among the works it cites.

Neural architecture search on imagenet in four gpu hours: A theoretically inspired perspective

Original

Chen, W., Gong, X., and Wang, Z. (2021) · 2021

Later among the works it cites.

Dnn-based topology optimisation: Spatial invariance and neural tangent kernel

Original

Dupuis, B. and Jacot, A. (2021) · 2021

Later among the works it cites.

Rapid training of deep neural networks without skip connections or normalization layers using deep kernel shaping

Original