Built on

B. Ghorbani, S. Mei, T. Misiakiewicz, and A. Montanari, “Linearized two-layers neural networks in high dimension”, arxiv:1904.12191
Original
1904
Earlier work this paper cites.
V. Vapnik, “The nature of statistical learning theory”, Springer science & business media, 1999
1999
Earlier work this paper cites.
A. Smola, Z. Ovári, and R. C. Williamson, “Regularization with dot-product kernels”, Advances in neural information processing systems
2000
Earlier work this paper cites.
L. Györfi, ed., “A distribution-free theory of nonparametric regression”, Springer, New York, 2002
2002
Earlier work this paper cites.
R. A. Adams and J. J. Fournier, “Sobolev spaces”, Elsevier, 2003
2003
Earlier work this paper cites.
H. Wendland, “Scattered data approximation”, Cambridge University Press, Cambridge, 2004
2004
Earlier work this paper cites.
A. Caponnetto and E. De Vito, “Optimal rates for the regularized least-squares algorithm”, Foundations of Computational Mathematics
2007
Earlier work this paper cites.
I. S. a. Andreas Christmann, “Support vector machines”, Springer-Verlag New York, New York, NY, first ed., 2008
2008
Earlier work this paper cites.
A. Bietti and F. Bach, “Deep equals shallow for ReLU networks in kernel regimes”, arXiv preprint arXiv:2009.14397
Original
2009
Earlier work this paper cites.
A. Caponnetto and Y. Yao, “Cross-validation based adaptation for regularization operators in learning theory”, Analysis and Applications
Original
2010
Earlier work this paper cites.

Similar

I. Steinwart and C. Scovel, “Mercer’s theorem on general domains: On the interaction between measures, kernels, and RKHSs”, Constructive Approximation
2012
Cited alongside, same era.
B. Neyshabur, R. Tomioka, and N. Srebro, “In search of the real inductive bias: On the role of implicit regularization in deep learning”, arxiv:1412.6614
Original
2015
Cited alongside, same era.
C. Zhang, S. Bengio, M. Hardt, B. Recht, and O. Vinyals, “Understanding deep learning requires rethinking generalization”, arxiv:1611.03530
Original
2017
Cited alongside, same era.
Curran Associates, Inc., 2018
A. Jacot, F. Gabriel, and C. Hongler, “Neural tangent kernel: Convergence and generalization in neural networks”, in “Advances in Neural Information Processing Systems”, S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett, eds., vol. 31 · 2018
Cited alongside, same era.
A. Rakhlin and X. Zhai, “Consistency of interpolation with Laplace kernels is a high-dimensional phenomenon”, arxiv:1812.11167

Then

S.-R. Fischer and I. Steinwart, “Sobolev norm learning rates for regularized least-squares algorithms”, Journal of Machine Learning Research
2020
Later among the works it cites.
T. Liang and A. Rakhlin, “Just interpolate: Kernel "ridgeless" regression can generalize”, The Annals of Statistics
Original
2020
Later among the works it cites.
PMLR, 2020
T. Liang, A. Rakhlin, and X. Zhai, “On the multiple descent of minimum-norm interpolants and restricted lower isometry of kernels”, in “Conference on Learning Theory”, pp. 2683–2711 · 2020
Later among the works it cites.
N. Ghosh, S. Mei, and B. Yu, “The three stages of learning dynamics in high-dimensional kernel methods”, arxiv:2111.07167
Original
2021
Later among the works it cites.
S. Mei and A. Montanari, “The generalization error of random features regression: Precise asymptotics and the double descent curve”, Communications on Pure and Applied Mathematics

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…

2023

Kernel interpolation generalizes poorly

Li, Yicheng, Zhang, Haobo, Lin, Qian

Understand

One of the most interesting problems in the recent renaissance of the studies in kernel regression might be whether the kernel interpolation can generalize well, since it may help us understand the `benign overfitting henomenon' reported in the literature on deep networks.

In this paper, under mild conditions, we show that for any $\varepsilon>0$, the generalization error of kernel interpolation is lower bounded by $\Omega(n^{-\varepsilon})$.
In other words, the kernel interpolation generalizes poorly for a large class of kernels.
As a direct corollary, we can show that overfitted wide neural networks defined on the sphere generalize poorly.

Built on

B. Ghorbani, S. Mei, T. Misiakiewicz, and A. Montanari, “Linearized two-layers neural networks in high dimension”, arxiv:1904.12191
Original
1904
Earlier work this paper cites.
V. Vapnik, “The nature of statistical learning theory”, Springer science & business media, 1999
1999
Earlier work this paper cites.
A. Smola, Z. Ovári, and R. C. Williamson, “Regularization with dot-product kernels”, Advances in neural information processing systems
2000
Earlier work this paper cites.
L. Györfi, ed., “A distribution-free theory of nonparametric regression”, Springer, New York, 2002
2002
Earlier work this paper cites.
R. A. Adams and J. J. Fournier, “Sobolev spaces”, Elsevier, 2003
2003
Earlier work this paper cites.
H. Wendland, “Scattered data approximation”, Cambridge University Press, Cambridge, 2004
2004
Earlier work this paper cites.
A. Caponnetto and E. De Vito, “Optimal rates for the regularized least-squares algorithm”, Foundations of Computational Mathematics
2007
Earlier work this paper cites.
I. S. a. Andreas Christmann, “Support vector machines”, Springer-Verlag New York, New York, NY, first ed., 2008
2008
Earlier work this paper cites.
A. Bietti and F. Bach, “Deep equals shallow for ReLU networks in kernel regimes”, arXiv preprint arXiv:2009.14397
Original
2009
Earlier work this paper cites.
A. Caponnetto and Y. Yao, “Cross-validation based adaptation for regularization operators in learning theory”, Analysis and Applications
Original
2010
Earlier work this paper cites.

Similar

I. Steinwart and C. Scovel, “Mercer’s theorem on general domains: On the interaction between measures, kernels, and RKHSs”, Constructive Approximation
2012
Cited alongside, same era.
B. Neyshabur, R. Tomioka, and N. Srebro, “In search of the real inductive bias: On the role of implicit regularization in deep learning”, arxiv:1412.6614
Original
2015
Cited alongside, same era.
C. Zhang, S. Bengio, M. Hardt, B. Recht, and O. Vinyals, “Understanding deep learning requires rethinking generalization”, arxiv:1611.03530
Original
2017
Cited alongside, same era.
Curran Associates, Inc., 2018
A. Jacot, F. Gabriel, and C. Hongler, “Neural tangent kernel: Convergence and generalization in neural networks”, in “Advances in Neural Information Processing Systems”, S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett, eds., vol. 31 · 2018
Cited alongside, same era.
A. Rakhlin and X. Zhai, “Consistency of interpolation with Laplace kernels is a high-dimensional phenomenon”, arxiv:1812.11167

Then

S.-R. Fischer and I. Steinwart, “Sobolev norm learning rates for regularized least-squares algorithms”, Journal of Machine Learning Research
2020
Later among the works it cites.
T. Liang and A. Rakhlin, “Just interpolate: Kernel "ridgeless" regression can generalize”, The Annals of Statistics
Original
2020
Later among the works it cites.
PMLR, 2020
T. Liang, A. Rakhlin, and X. Zhai, “On the multiple descent of minimum-norm interpolants and restricted lower isometry of kernels”, in “Conference on Learning Theory”, pp. 2683–2711 · 2020
Later among the works it cites.
N. Ghosh, S. Mei, and B. Yu, “The three stages of learning dynamics in high-dimensional kernel methods”, arxiv:2111.07167
Original
2021
Later among the works it cites.
S. Mei and A. Montanari, “The generalization error of random features regression: Precise asymptotics and the double descent curve”, Communications on Pure and Applied Mathematics

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Original

2018

Cited alongside, same era.

G. Blanchard and N. Mücke, “Optimal rates for regularization of statistical inverse learning problems”, Foundations of Computational Mathematics

2018

Cited alongside, same era.

M. Belkin, D. Hsu, S. Ma, and S. Mandal, “Reconciling modern machine-learning practice and the classical bias–variance trade-off”, Proceedings of the National Academy of Sciences

2019

Cited alongside, same era.

Z. Allen-Zhu, Y. Li, and Z. Song, “A convergence theory for deep learning via over-parameterization”, 1811.03962

2019

Cited alongside, same era.

M. J. Wainwright, “High-dimensional statistics: A non-asymptotic viewpoint”, Cambridge University Press, 2019

2019

Cited alongside, same era.

Curran Associates, Inc., 2019

J. Lee, L. Xiao, S. Schoenholz, Y. Bahri, R. Novak, J. Sohl-Dickstein, and J. Pennington, “Wide neural networks of any depth evolve as linear models under gradient descent”, in “Advances in Neural Information Processing Systems”, vol. 32 · 2019

Cited alongside, same era.

2022

Later among the works it cites.

PMLR, 2022

S. Buchholz, “Kernel interpolation in Sobolev spaces is not consistent in low dimensions”, in “Proceedings of Thirty Fifth Conference on Learning Theory”, P.-L. Loh and M. Raginsky, eds., vol. 178 of Proceedings of Machine Learning Research · 2022

Later among the works it cites.

D. Beaglehole, M. Belkin, and P. Pandit, “Kernel ridgeless regression is inconsistent in low dimensions”, arxiv:2205.13525

Original

2022

Later among the works it cites.

H. Zhang, Y. Li, and Q. Lin, “On the optimality of misspecified spectral algorithms”, arxiv:2303.14942

Original

2023

Closest in time.

J. Lai, M. Xu, R. Chen, and Q. Lin, “Generalization ability of wide neural networks on R”, arxiv:2302.05933

Original

2023

Closest in time.