Contrasting random and learned features in deep Bayesian linear regression

A. Canatar, B. Bordelon, and C. Pehlevan, Spectral bias and task-model alignment explain generalization in kernel regression and infinitely wide neural networks, Nature Communications 12

Original

2006

Cited alongside, same era.

S. d’Ascoli, L. Sagun, and G. Biroli, Triple descent and the two kinds of overfitting: where and why do they appear?, Journal of Statistical Mechanics: Theory and Experiment 2021

Original

2006

Cited alongside, same era.

K. Aitken and G. Gur-Ari, On the asymptotics of wide networks with polynomial activations, arXiv preprint arXiv:2006.06687 (2020), arXiv:2006.06687

Original

2006

Cited alongside, same era.

C. Mingard, G. Valle-Pérez, J. Skalse, and A. A. Louis, Is SGD a Bayesian sampler? well, almost, Journal of Machine Learning Research 22

Original

2006

Cited alongside, same era.

J. Lee, S. Schoenholz, J. Pennington, B. Adlam, L. Xiao, R. Novak, and J. Sohl-Dickstein, Finite versus infinite neural networks: an empirical study, in Advances in Neural Information Processing Systems , Vol. 33 (2020) pp. 15156–15172, arXiv:2007.15801

Original

2007

Cited alongside, same era.

J. A. Zavatone-Veth and C. Pehlevan, Activation function dependence of the storage capacity of treelike neural networks, Physical Review E 103

Original

2007

Cited alongside, same era.

B. He, B. Lakshminarayanan, and Y. W. Teh, Bayesian deep ensembles via the neural tangent kernel, in Advances in Neural Information Processing Systems , Vol. 33, edited by H. Larochelle, M. Ranzato, R. Hadsell, M. Balcan, and H. Lin (Curran Associates, Inc., 2020) pp. 1010–1022, arXiv:2007.05864

Original

2007

Cited alongside, same era.

J. D. Hunter, Matplotlib: A 2d graphics environment, Computing in Science & Engineering 9

Original

2007

Cited alongside, same era.

A. Jacot, F. Gabriel, and C. Hongler, Neural tangent kernel: Convergence and generalization in neural networks, in Advances in Neural Information Processing Systems , Vol. 31, edited by S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett (2018) arXiv:1806.07572

Original

2018

Later among the works it cites.

M. Belkin, D. Hsu, S. Ma, and S. Mandal, Reconciling modern machine-learning practice and the classical bias–variance trade-off, Proceedings of the National Academy of Sciences 116

Original

2019

Later among the works it cites.

J. R. Magnus and H. Neudecker, Matrix differential calculus with applications in statistics and econometrics (John Wiley & Sons, 2019)

2019

Later among the works it cites.

M. S. Advani, A. M. Saxe, and H. Sompolinsky, High-dimensional dynamics of generalization error in neural networks, Neural Networks 132

Original

2020

Later among the works it cites.

C. R. Harris, K. J. Millman, S. J. van der Walt, R. Gommers, P. Virtanen, D. Cournapeau, E. Wieser, J. Taylor, S. Berg, N. J. Smith, R. Kern, M. Picus, S. Hoyer, M. H. van Kerkwijk, M. Brett, A. Haldane, J. Fernández del Río, M. Wiebe, P. Peterson, P. Gérard-Marchant, K. Sheppard, T. Reddy, W. Weckesser, H. Abbasi, C. Gohlke, and T. E. Oliphant, Array programming with NumPy, Nature 585

2020

Later among the works it cites.

P. Virtanen, R. Gommers, T. E. Oliphant, M. Haberland, T. Reddy, D. Cournapeau, E. Burovski, P. Peterson, W. Weckesser, J. Bright, S. J. van der Walt, M. Brett, J. Wilson, K. J. Millman, N. Mayorov, A. R. J. Nelson, E. Jones, R. Kern, E. Larson, C. J. Carey, İ. Polat, Y. Feng, E. W. Moore, J. VanderPlas, D. Laxalde, J. Perktold, R. Cimrman, I. Henriksen, E. A. Quintero, C. R. Harris, A. M. Archibald, A. H. Ribeiro, F. Pedregosa, P. van Mulbregt, and SciPy 1.0 Contributors, SciPy 1.0: Fundamental Algorithms for Scientific Computing in Python, Nature Methods 17

2020

Later among the works it cites.

C. Zhang, S. Bengio, M. Hardt, B. Recht, and O. Vinyals, Understanding deep learning (still) requires rethinking generalization, Communications of the ACM 64

Original

2021

Later among the works it cites.

M. Refinetti, S. Goldt, F. Krzakala, and L. Zdeborova, Classifying high-dimensional Gaussian mixtures: Where kernel methods fail and neural networks succeed, in Proceedings of the 38th International Conference on Machine Learning , Proceedings of Machine Learning Research, Vol. 139, edited by M. Meila and T. Zhang (PMLR, 2021) pp. 8936–8947, arXiv:2102.11742

Original

2021

Later among the works it cites.

B. Loureiro, C. Gerbelot, H. Cui, S. Goldt, F. Krzakala, M. Mezard, and L. Zdeborova, Learning curves of generic features maps for realistic datasets with a teacher-student model, in Advances in Neural Information Processing Systems , Vol. 34, edited by A. Beygelzimer, Y. Dauphin, P. Liang, and J. W. Vaughan (2021) arXiv:2102.08127

Original

2021

Later among the works it cites.

J. A. Zavatone-Veth and C. Pehlevan, Exact marginal prior distributions of finite Bayesian neural networks, in Advances in Neural Information Processing Systems , Vol. 34, edited by A. Beygelzimer, Y. Dauphin, P. Liang, and J. W. Vaughan (2021) arXiv:2104.11734

Original

2021

Later among the works it cites.

K. T. Grosvenor and R. Jefferson, The edge of chaos: quantum field theory and deep neural networks, arXiv preprint arXiv:2109.13247 (2021), arXiv:2109.13247

Original

2021

Later among the works it cites.

J. A. Zavatone-Veth, A. Canatar, B. S. Ruben, and C. Pehlevan, Asymptotics of representation learning in finite Bayesian neural networks, in Advances in Neural Information Processing Systems , Vol. 34, edited by A. Beygelzimer, Y. Dauphin, P. Liang, and J. W. Vaughan (2021) arXiv:2106.00651

Original

2021

Later among the works it cites.

J. A. Zavatone-Veth and C. Pehlevan, Depth induces scale-averaging in overparameterized linear Bayesian neural networks, in Asilomar Conference on Signals, Systems, and Computers , Vol. 55 (2021) arXiv:2111.11954

Original

2021

Later among the works it cites.

G. Naveh and Z. Ringel, A self consistent theory of Gaussian processes captures feature learning effects in finite CNNs, in Advances in Neural Information Processing Systems , Vol. 34, edited by A. Beygelzimer, Y. Dauphin, P. Liang, and J. W. Vaughan (2021) arXiv:2106.04110

Original

2021

Later among the works it cites.

P. Izmailov, S. Vikram, M. D. Hoffman, and A. G. G. Wilson, What are Bayesian neural network posteriors really like?, in Proceedings of the 38th International Conference on Machine Learning , Proceedings of Machine Learning Research, Vol. 139, edited by M. Meila and T. Zhang (PMLR, 2021) pp. 4629–4640, arXiv:2104.14421

Original

2021

Later among the works it cites.

M. Cranmer, D. Tamayo, H. Rein, P. Battaglia, S. Hadden, P. J. Armitage, S. Ho, and D. N. Spergel, A Bayesian neural network predicts the dissolution of compact planetary systems, Proceedings of the National Academy of Sciences 118

Original

2021

Later among the works it cites.

DLMF, NIST Digital Library of Mathematical Functions , http://dlmf.nist.gov/, Release 1.1.1 of 2021-03-15 (2021), f. W. J. Olver, A. B. Olde Daalhuis, D. W. Lozier, B. I. Schneider, R. F. Boisvert, C. W. Clark, B. R. Miller, B. V. Saunders, H. S. Cohl, and M. A. McClain, eds

2021

Later among the works it cites.

H. Jin, P. K. Banerjee, and G. Montufar, Learning curves for Gaussian process regression with power-law priors and targets, in International Conference on Learning Representations (2022) arXiv:2110.12231

Original

2022

Closest in time.

D. A. Roberts, S. Yaida, and B. Hanin, The Principles of Deep Learning Theory: An Effective Theory Approach to Understanding Neural Networks (Cambridge University Press, 2022) arXiv:2106.10165

Original

2022

Closest in time.

J. W. Rocks and P. Mehta, Bias-variance decomposition of overparameterized regression with random linear features, arXiv preprint arXiv:2203.05443 (2022), arXiv:2203.05443

Original

2022

Closest in time.