2024

A Dynamical Model of Neural Scaling Laws

Bordelon, Blake, Atanasov, Alexander, Pehlevan, Cengiz

Understand

On a variety of tasks, the performance of neural networks predictably improves with training time, dataset size and model size across many orders of magnitude.

This phenomenon is known as a neural scaling law.
Of fundamental importance is the compute-optimal scaling law, which reports the performance as a function of units of compute when choosing model sizes optimally.
We analyze a random feature model trained with gradient descent as a solvable model of network training and generalization.

Built on

Statistical dynamics of classical systems
Martin, P. C., Siggia, E., and Rose, H · 1973
Earlier work this paper cites.
Dynamic theory of the spin-glass phase
Sompolinsky, H. and Zippelius, A · 1981
Earlier work this paper cites.
Fast rates for regularized least-squares algorithm
Caponnetto, A. and Vito, E. D · 2005
Earlier work this paper cites.
Algorithms for learning kernels based on centered alignment
Cortes, C., Mohri, M., and Rostamizadeh, A · 2012
Earlier work this paper cites.
Wide residual networks
Original
Zagoruyko, S. and Komodakis, N · 2016
Earlier work this paper cites.
Deep learning scaling is predictable, empirically
Original
Hestness, J., Narang, S., Ardalani, N., Diamos, G., Jun, H., Kianinejad, H., Patwary, M. M. A., Yang, Y., and Zhou, Y · 2017
Earlier work this paper cites.
Path integral approach to random neural networks
Crisanti, A. and Sompolinsky, H · 2018
Earlier work this paper cites.
Neural tangent kernel: Convergence and generalization in neural networks
Jacot, A., Gabriel, F., and Hongler, C · 2018
Earlier work this paper cites.
On lazy training in differentiable programming
Chizat, L., Oyallon, E., and Bach, F · 2019
Earlier work this paper cites.
Passed & spurious: Descent algorithms and local minima in spiked matrix-tensor models
Mannelli, S. S., Krzakala, F., Urbani, P., and Zdeborova, L · 2019
Earlier work this paper cites.
High-dimensional dynamics of generalization error in neural networks
Advani, M. S., Saxe, A. M., and Sompolinsky, H · 2020
Earlier work this paper cites.
Spectrum dependent learning curves in kernel regression and wide neural networks
Bordelon, B., Canatar, A., and Pehlevan, C · 2020
Earlier work this paper cites.
Asymptotics of wide networks from feynman diagrams
Dyer, E. and Gur-Ari, G · 2020
Earlier work this paper cites.
Double trouble in double descent: Bias and variance (s) in the lazy regime
d’Ascoli, S., Refinetti, M., Biroli, G., and Krzakala, F · 2020
Earlier work this paper cites.
Deep learning versus kernel learning: an empirical study of loss landscape geometry and the time evolution of the neural tangent kernel
Fort, S., Dziugaite, G. K., Paul, M., Kharaghani, S., Roy, D. M., and Ganguli, S · 2020
Earlier work this paper cites.
Scaling description of generalization with number of parameters in deep learning
Geiger, M., Jacot, A., Spigler, S., Gabriel, F., Sagun, L., d’Ascoli, S., Biroli, G., Hongler, C., and Wyart, M · 2020
Earlier work this paper cites.
Statistical field theory for neural networks , volume 970
Helias, M. and Dahmen, D · 2020
Earlier work this paper cites.
Scaling laws for neural language models
Original
Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., and Amodei, D · 2020
Earlier work this paper cites.
Dynamical mean-field theory for stochastic gradient descent in gaussian mixture classification
Mignacco, F., Krzakala, F., Urbani, P., and Zdeborová, L · 2020
Earlier work this paper cites.

Similar

Asymptotic learning curves of kernel methods: empirical data versus teacher–student paradigm
Spigler, S., Geiger, M., and Wyart, M · 2020
Cited alongside, same era.
Explaining neural scaling laws
Original
Bahri, Y., Dyer, E., Kaplan, J., Lee, J., and Sharma, U · 2021
Cited alongside, same era.
Model, sample, and epoch-wise descents: exact solution of gradient flow in the random feature model
Bodin, A. and Macris, N · 2021
Cited alongside, same era.
Spectral bias and task-model alignment explain generalization in kernel regression and infinitely wide neural networks
Canatar, A., Bordelon, B., and Pehlevan, C · 2021
Cited alongside, same era.
Generalization error rates in kernel regression: The crossover from the noiseless to noisy regime
Cui, H., Loureiro, B., Krzakala, F., and Zdeborová, L · 2021
Cited alongside, same era.
Properties of the after kernel
Original
Long, P. M · 2021
Cited alongside, same era.
Learning curves of generic features maps for realistic datasets with a teacher-student model
Loureiro, B., Gerbelot, C., Cui, H., Goldt, S., Krzakala, F., Mezard, M., and Zdeborová, L · 2021
Cited alongside, same era.
Sgd in the large: Average-case analysis, asymptotics, and stepsize criticality
Paquette, C., Lee, K., Pedregosa, F., and Paquette, E · 2021
Cited alongside, same era.
The eigenlearning framework: A conservation law perspective on kernel regression and wide neural networks
Original
Simon, J. B., Dickens, M., Karkada, D., and DeWeese, M. R · 2021
Cited alongside, same era.
Last iterate convergence of sgd for least-squares in the interpolation regime
Varre, A. V., Pillaud-Vivien, L., and Flammarion, N · 2021
Cited alongside, same era.
Tuning large neural networks via zero-shot hyperparameter transfer
Yang, G., Hu, E. J., Babuschkin, I., Sidor, S., Liu, X., Farhi, D., Ryder, N., Pachocki, J., Chen, W., and Gao, J · 2021
Cited alongside, same era.
Neural networks as kernel learners: The silent alignment effect
Atanasov, A., Bordelon, B., and Pehlevan, C · 2022
Cited alongside, same era.

Then

A solvable model of neural scaling laws
Original
Maloney, A., Roberts, D. A., and Sully, J · 2022
Later among the works it cites.
The generalization error of random features regression: Precise asymptotics and the double descent curve
Mei, S. and Montanari, A · 2022
Later among the works it cites.
The effective noise of stochastic gradient descent
Mignacco, F. and Urbani, P · 2022
Later among the works it cites.
The principles of deep learning theory
Roberts, D. A., Yaida, S., and Hanin, B · 2022
Later among the works it cites.
Beyond neural scaling laws: beating power law scaling via data pruning
Sorscher, B., Geirhos, R., Shekhar, S., Ganguli, S., and Morcos, A · 2022
Later among the works it cites.
Limitations of the ntk for understanding generalization in deep learning
Original
Vyas, N., Bansal, Y., and Nakkiran, P · 2022
Later among the works it cites.
Contrasting random and learned features in deep bayesian linear regression
Zavatone-Veth, J. A., Tong, W. L., and Pehlevan, C · 2022
Later among the works it cites.
Getting vit in shape: Scaling laws for compute-optimal model design
Original
Alabdulmohsin, I., Zhai, X., Kolesnikov, A., and Beyer, L · 2023
Later among the works it cites.
A theory for emergence of complex skills in language models
Original
Arora, S. and Goyal, A · 2023
Later among the works it cites.
The onset of variance-limited behavior for networks in the lazy and rich regimes
Atanasov, A., Bordelon, B., Sainathan, S., and Pehlevan, C · 2023
Later among the works it cites.
Dynamics of finite width kernel and prediction fluctuations in mean field neural networks
Original
Bordelon, B. and Pehlevan, C · 2023
Later among the works it cites.
Depthwise hyperparameter transfer in residual networks: Dynamics and scaling limit, 2023
Bordelon, B., Noci, L., Li, M. B., Hanin, B., and Pehlevan, C · 2023
Later among the works it cites.
The quantization model of neural scaling
Original
Michaud, E. J., Liu, Z., Girit, U., and Tegmark, M · 2023
Later among the works it cites.
Scaling data-constrained language models
Original
Muennighoff, N., Rush, A. M., Barak, B., Scao, T. L., Piktus, A., Tazi, N., Pyysalo, S., Wolf, T., and Raffel, C · 2023
Later among the works it cites.
Learning curves for noisy heterogeneous feature-subsampled ridge ensembles
Ruben, B. S. and Pehlevan, C · 2023
Later among the works it cites.
More is better in modern machine learning: when infinite overparameterization is optimal and overfitting is obligatory
Original
Simon, J. B., Karkada, D., Ghosh, N., and Belkin, M · 2023
Later among the works it cites.
Feature-learning networks are consistent across widths at realistic scales
Original
Vyas, N., Atanasov, A., Bordelon, B., Morwani, D., Sainathan, S., and Pehlevan, C · 2023
Later among the works it cites.
Learning curves for deep structured gaussian feature models, 2023
Zavatone-Veth, J. A. and Pehlevan, C · 2023
Later among the works it cites.
Scaling mlps: A tale of inductive bias
Bachmann, G., Anagnostidis, S., and Hofmann, T · 2024
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…