2022

Machine Learning Model Sizes and the Parameter Gap

Villalobos, Pablo, Sevilla, Jaime, Besiroglu, Tamay et al.

Understand

We study trends in model size of notable machine learning systems over time using a curated dataset.

From 1950 to 2018, model size in language models increased steadily by seven orders of magnitude.
The trend then accelerated, with model size increasing by another five orders of magnitude in just 4 years from 2018 to 2022.
Vision models grew at a more constant pace, totaling 7 orders of magnitude of growth between 1950 and 2022.

Built on

J. Kaplan et al. , “Scaling Laws for Neural Language Models,” 2020, _eprint: 2001.08361
2001
Earlier work this paper cites.
O. Sharir, B. Peleg, and Y. Shoham, “The Cost of Training NLP Models: A Concise Overview,” 2020, _eprint: 2004.08900
2004
Earlier work this paper cites.
T.B. Brown et al. , “Language Models are Few-Shot Learners,” 2020, _eprint: 2005.14165
2005
Earlier work this paper cites.
N.C. Thompson et al. , “The Computational Limits of Deep Learning,” 2020, _eprint: 2007.05558
2007
Earlier work this paper cites.
D. Amodei and D. Hernandez, “ AI and Compute ,” May 2018, published: OpenAI Blog
2018
Earlier work this paper cites.
M. Naumov et al. , “Deep Learning Recommendation Model for Personalization and Recommendation Systems,” May 2019
2019
Earlier work this paper cites.
M. Shoeybi et al. , “Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism,” Sep. 2019
2019
Earlier work this paper cites.
Z. Lan et al. , “ALBERT: A Lite BERT for Self-supervised Learning of Language Representations,” Sep. 2019
2019
Earlier work this paper cites.
D. Lepikhin et al. , “Gshard: Scaling giant models with conditional computation and automatic sharding,” 2020
2020
Earlier work this paper cites.
N. Ahmed and M. Wahed, “The de-democratization of ai: Deep learning and the compute divide in artificial intelligence research,” 2020
2020
Earlier work this paper cites.
C. Rosset, “Turing-NLG: A 17-billion-parameter language model by Microsoft,” Feb. 2020
2020
Earlier work this paper cites.
Z. Zhang et al. , “CPM: A Large-scale Generative Chinese Pre-trained Language Model,” Dec. 2020
2020
Earlier work this paper cites.
W. Antoun, F. Baly, and H. Hajj, “AraGPT2: Pre-Trained Transformer for Arabic Language Generation,” Dec. 2020
2020
Earlier work this paper cites.

Similar

H. Pham et al. , “Meta Pseudo Labels,” Mar. 2020
2020
Cited alongside, same era.
J. Sevilla, P. Villalobos, and J.F. Cerón, “\emphParameter Counts in Machine Learning,” Jun. 2021, published: Alignment Forum (blog)
2021
Cited alongside, same era.
W. Fedus, B. Zoph, and N. Shazeer, “Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity,” Jan. 2021
2021
Cited alongside, same era.
A. Askell et al. , “A general language assistant as a laboratory for alignment,” 2021
2021
Cited alongside, same era.
D. Narayanan et al. , “Efficient large-scale language model training on gpu clusters using megatron-lm,” 2021
2021
Cited alongside, same era.
S. Rajbhandari et al. , “Zero-infinity: Breaking the gpu memory wall for extreme scale deep learning,” 2021
2021
Cited alongside, same era.
D. Patterson et al. , “Carbon emissions and large neural network training,” 2021
2021
Cited alongside, same era.
D. Hendrycks et al. , “Measuring mathematical problem solving with the math dataset,” 2021
2021
Cited alongside, same era.
A.L. Jones, “Scaling Scaling Laws with Board Games,” arXiv preprint arXiv:2104.03113 , pp. 1–8, 2021
Original
2021
Cited alongside, same era.
D. Mudigere et al. , “Software-Hardware Co-design for Fast and Scalable Training of Deep Learning Recommendation Models,” Apr. 2021
2021
Cited alongside, same era.
D.R. So et al. , “Primer: Searching for Efficient Transformers for Language Modeling,” Sep. 2021
2021
Cited alongside, same era.
W. Zeng et al. , “PanGu-$\alpha$: Large-scale Autoregressive Pretrained Chinese Language Models with Auto-parallel Computation,” Apr. 2021
2021
Cited alongside, same era.

Then

W.N. Hsu et al. , “HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units,” Jun. 2021
2021
Later among the works it cites.
J. Lin et al. , “M6-10T: A Sharing-Delinking Paradigm for Efficient Multi-Trillion Parameter Pretraining,” Oct. 2021
2021
Later among the works it cites.
A. Alvi and P. Kharya, “Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, the World’s Largest and Most Powerful Generative Language Model,” Oct. 2021
2021
Later among the works it cites.
S. Wu et al. , “Yuan 1.0: Large-Scale Pre-trained Language Model in Zero-Shot and Few-Shot Learning,” Oct. 2021
2021
Later among the works it cites.
J. Hoffmann et al. , “Training compute-optimal large language models,” 2022
2022
Closest in time.
Z. Talat et al. , “You reap what you sow: On the challenges of bias evaluation under multilingual settings,” in Proceedings of BigScience Episode #5 – Workshop on Challenges & Perspectives in Creating Large Language Models . virtual+Dublin: Association for Computational Linguistics, May 2022, pp. 26–41
2022
Closest in time.
J. Sevilla et al. , “Compute trends across three eras of machine learning,” https://epochai.org/blog/compute-trends , 2022, accessed: 2022-06-09
2022
Closest in time.
D. Jones, “Growth in flops used to train ml models,” 2022
2022
Closest in time.
S. Zhang et al. , “Opt: Open pre-trained transformer language models,” 2022
2022
Closest in time.
R. Thoppilan et al. , “LaMDA: Language Models for Dialog Applications,” Jan. 2022
2022
Closest in time.
C. Leahy, “Announcing GPT-NeoX-20B,” Feb. 2022
2022
Closest in time.
A. Chowdhery et al. , “Palm: Scaling language modeling with pathways,” 2022
2022
Closest in time.
J.B. Alayrac et al. , “Flamingo: a visual language model for few-shot learning,” 2022
2022
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…