Built on

K. Pearson, “On lines and planes of closest fit to systems of points in space,” Philos. Mag
1901
Earlier work this paper cites.
S. Spigler, M. Geiger, and M. Wyart, “Asymptotic learning curves of kernel methods: empirical data versus teacher–student paradigm,” Journal of Statistical Mechanics: Theory and Experiment
Original
1905
Earlier work this paper cites.
T. Wolf, L. Debut, V. Sanh, J. Chaumond, C. Delangue, A. Moi, P. Cistac, T. Rault, R. Louf, M. Funtowicz, et al
Original
1910
Earlier work this paper cites.
P. Nakkiran, G. Kaplun, Y. Bansal, T. Yang, B. Barak, and I. Sutskever, “Deep double descent: Where bigger models and more data hurt,” Journal of Statistical Mechanics: Theory and Experiment
Original
1912
Earlier work this paper cites.
F. J. Dyson, “The S S Matrix in Quantum Electrodynamics,” Phys. Rev
1949
Earlier work this paper cites.
J. Schwinger, “On the Green’s functions of quantized fields. I,” Proceedings of the National Academy of Sciences
1951
Earlier work this paper cites.
V. A. Marčenko and L. A. Pastur, “Distribution of Eigenvalues for Some Sets of Random Matrices,” Mathematics of the USSR-Sbornik
1967
Earlier work this paper cites.
G. ’t Hooft, “A Planar Diagram Theory for Strong Interactions,” Nucl. Phys
1974
Earlier work this paper cites.
J. W. Silverstein and Z. D. Bai, “On the Empirical Distribution of Eigenvalues of a Class of Large Dimensional Random Matrices,” Journal of Multivariate Analysis
1995
Earlier work this paper cites.
M. Opper, “Statistical mechanics of learning: Generalization,” The handbook of brain theory and neural networks
1995
Earlier work this paper cites.
Springer, 1996
R. M. Neal, “Priors for infinite networks,” in Bayesian Learning for Neural Networks · 1996
Earlier work this paper cites.
J. Kaplan, S. McCandlish, T. Henighan, T. B. Brown, B. Chess, R. Child, S. Gray, A. Radford, J. Wu, and D. Amodei, “Scaling laws for neural language models,” arXiv:2001.08361 [cs.LG]
Original
2001
Earlier work this paper cites.
M. Opper, “Learning to generalize,” Frontiers of Life
2001
Earlier work this paper cites.
https://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.3.4934&rep=rep1&type=pdf
J. Friedman and B. E. Popescu, “Gradient directed regularization for linear regression and classification,” tech. rep., 2002 · 2002
Earlier work this paper cites.

Similar

U. Sharma and J. Kaplan, “Scaling Laws from the Data Manifold Dimension,” Journal of Machine Learning Research
Original
2004
Cited alongside, same era.
M. Mitzenmacher, “A brief history of generative models for power law and lognormal distributions,” Internet mathematics
2004
Cited alongside, same era.
Z. Burda, A. Görlich, A. Jarosz, and J. Jurkiewicz, “Signal and noise in correlation matrix,” Physica A: Statistical Mechanics and its Applications
2004
Cited alongside, same era.
MIT Press, Cambridge, MA, USA, 2004
E. Levina and P. J. Bickel, “Maximum Likelihood Estimation of Intrinsic Dimension,” in Proceedings of the 17th International Conference on Neural Information Processing Systems · 2004
Cited alongside, same era.
T. B. Brown, B. P. Mann, N. Ryder, M. Subbiah, J. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, S. Agarwal, A. Herbert-Voss, G. Krüger, T. Henighan, R. Child, A. Ramesh, D. M. Ziegler, J. Wu, C. Winter, C. Hesse, M. Chen, E. J. Sigler, M. Litwin, S. Gray, B. Chess, J. Clark, C. Berner, S. McCandlish, A. Radford, I. Sutskever, and D. Amodei, “Language Models are Few-Shot Learners,” arXiv:2005.14165 [cs.CL]

Then

S. Bradde and W. Bialek, “PCA meets RG,” Journal of Statistical Physics
Original
2017
Later among the works it cites.
Springer, 2017
J. A. Mingo and R. Speicher, Free Probability and Random Matrices , vol. 35 · 2017
Later among the works it cites.
E. Facco, M. d’Errico, A. Rodriguez, and A. Laio, “Estimating the intrinsic dimension of datasets by a minimal neighborhood information,” Scientific Reports
2017
Later among the works it cites.
C. Louart, Z. Liao, and R. Couillet, “A Random Matrix Approach to Neural Networks,” The Annals of Applied Probability
Original
2018
Later among the works it cites.
M. Belkin, D. Hsu, S. Ma, and S. Mandal, “Reconciling modern machine-learning practice and the classical bias-variance trade-off,” Proceedings of the National Academy of Sciences
Original
2019

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…

2022

A Solvable Model of Neural Scaling Laws

Maloney, Alexander, Roberts, Daniel A., Sully, James

Understand

Large language models with a huge number of parameters, when trained on near internet-sized number of tokens, have been empirically shown to obey neural scaling laws: specifically, their performance behaves predictably as a power law in either parameters or dataset size until bottlenecked by the other resource.

To understand this better, we first identify the necessary properties allowing such scaling laws to arise and then propose a statistical model -- a joint generative data model and random feature model -- that captures this neural scaling phenomenology.
By solving this model in the dual limit of large training set size and large number of parameters, we gain insight into (i) the statistical structure of datasets and tasks that lead to scaling laws, (ii) the way nonlinear feature maps, such as those provided by neural networks, enable scaling laws when trained on these datasets, (iii) the optimality of the equiparameterization scaling of training sets and parameters, and (iv) whether such scaling laws can break down and how they behave when they do.
Key findings are the manner in which the power laws that occur in the statistics of natural datasets are extended by nonlinear random feature maps and then translated into power-law scalings of the test loss and how the finite extent of the data's spectral power law causes the model's performance to plateau.

Built on

K. Pearson, “On lines and planes of closest fit to systems of points in space,” Philos. Mag
1901
Earlier work this paper cites.
S. Spigler, M. Geiger, and M. Wyart, “Asymptotic learning curves of kernel methods: empirical data versus teacher–student paradigm,” Journal of Statistical Mechanics: Theory and Experiment
Original
1905
Earlier work this paper cites.
T. Wolf, L. Debut, V. Sanh, J. Chaumond, C. Delangue, A. Moi, P. Cistac, T. Rault, R. Louf, M. Funtowicz, et al
Original
1910
Earlier work this paper cites.
P. Nakkiran, G. Kaplun, Y. Bansal, T. Yang, B. Barak, and I. Sutskever, “Deep double descent: Where bigger models and more data hurt,” Journal of Statistical Mechanics: Theory and Experiment
Original
1912
Earlier work this paper cites.
F. J. Dyson, “The S S Matrix in Quantum Electrodynamics,” Phys. Rev
1949
Earlier work this paper cites.
J. Schwinger, “On the Green’s functions of quantized fields. I,” Proceedings of the National Academy of Sciences
1951
Earlier work this paper cites.
V. A. Marčenko and L. A. Pastur, “Distribution of Eigenvalues for Some Sets of Random Matrices,” Mathematics of the USSR-Sbornik
1967
Earlier work this paper cites.
G. ’t Hooft, “A Planar Diagram Theory for Strong Interactions,” Nucl. Phys
1974
Earlier work this paper cites.
J. W. Silverstein and Z. D. Bai, “On the Empirical Distribution of Eigenvalues of a Class of Large Dimensional Random Matrices,” Journal of Multivariate Analysis
1995
Earlier work this paper cites.
M. Opper, “Statistical mechanics of learning: Generalization,” The handbook of brain theory and neural networks
1995
Earlier work this paper cites.
Springer, 1996
R. M. Neal, “Priors for infinite networks,” in Bayesian Learning for Neural Networks · 1996
Earlier work this paper cites.
J. Kaplan, S. McCandlish, T. Henighan, T. B. Brown, B. Chess, R. Child, S. Gray, A. Radford, J. Wu, and D. Amodei, “Scaling laws for neural language models,” arXiv:2001.08361 [cs.LG]
Original
2001
Earlier work this paper cites.
M. Opper, “Learning to generalize,” Frontiers of Life
2001
Earlier work this paper cites.
https://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.3.4934&rep=rep1&type=pdf
J. Friedman and B. E. Popescu, “Gradient directed regularization for linear regression and classification,” tech. rep., 2002 · 2002
Earlier work this paper cites.

Similar

U. Sharma and J. Kaplan, “Scaling Laws from the Data Manifold Dimension,” Journal of Machine Learning Research
Original
2004
Cited alongside, same era.
M. Mitzenmacher, “A brief history of generative models for power law and lognormal distributions,” Internet mathematics
2004
Cited alongside, same era.
Z. Burda, A. Görlich, A. Jarosz, and J. Jurkiewicz, “Signal and noise in correlation matrix,” Physica A: Statistical Mechanics and its Applications
2004
Cited alongside, same era.
MIT Press, Cambridge, MA, USA, 2004
E. Levina and P. J. Bickel, “Maximum Likelihood Estimation of Intrinsic Dimension,” in Proceedings of the 17th International Conference on Neural Information Processing Systems · 2004
Cited alongside, same era.
T. B. Brown, B. P. Mann, N. Ryder, M. Subbiah, J. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, S. Agarwal, A. Herbert-Voss, G. Krüger, T. Henighan, R. Child, A. Ramesh, D. M. Ziegler, J. Wu, C. Winter, C. Hesse, M. Chen, E. J. Sigler, M. Litwin, S. Gray, B. Chess, J. Clark, C. Berner, S. McCandlish, A. Radford, I. Sutskever, and D. Amodei, “Language Models are Few-Shot Learners,” arXiv:2005.14165 [cs.CL]

Then

S. Bradde and W. Bialek, “PCA meets RG,” Journal of Statistical Physics
Original
2017
Later among the works it cites.
Springer, 2017
J. A. Mingo and R. Speicher, Free Probability and Random Matrices , vol. 35 · 2017
Later among the works it cites.
E. Facco, M. d’Errico, A. Rodriguez, and A. Laio, “Estimating the intrinsic dimension of datasets by a minimal neighborhood information,” Scientific Reports
2017
Later among the works it cites.
C. Louart, Z. Liao, and R. Couillet, “A Random Matrix Approach to Neural Networks,” The Annals of Applied Probability
Original
2018
Later among the works it cites.
M. Belkin, D. Hsu, S. Ma, and S. Mandal, “Reconciling modern machine-learning practice and the classical bias-variance trade-off,” Proceedings of the National Academy of Sciences
Original
2019

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Original

2005

Cited alongside, same era.

A. Canatar, B. Bordelon, and C. Pehlevan, “Spectral Bias and Task-Model Alignment Explain Generalization in Kernel Regression and Infinitely Wide Neural Networks,” Nature Communications

Original

2006

Cited alongside, same era.

Springer Science & Business Media, 2009

T. Hastie, R. Tibshirani, and J. Friedman, The Elements of Statistical Learning · 2009

Cited alongside, same era.

A. Clauset, C. R. Shalizi, and M. E. J. Newman, “Power-Law Distributions in Empirical Data,” SIAM Review

Original

2009

Cited alongside, same era.

https://www.cs.toronto.edu/~kriz/cifar.html

A. Krizhevsky, “Learning multiple layers of features from tiny images,” 2009 · 2009

Cited alongside, same era.

T. Henighan, J. Kaplan, M. Katz, M. Chen, C. Hesse, J. Jackson, H. Jun, T. B. Brown, P. Dhariwal, S. Gray, et al

Original

2010

Cited alongside, same era.

A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, et al

Original

2010

Cited alongside, same era.

G. Raskutti, M. J. Wainwright, and B. Yu, “Early stopping and non-parametric regression: an optimal data-dependent stopping rule,” The Journal of Machine Learning Research

Original

2014

Cited alongside, same era.