2020

The Cost of Training NLP Models: A Concise Overview

Sharir, Or, Peleg, Barak, Shoham, Yoav

Understand

We review the cost of training large-scale language models, and the drivers of these costs.

The intended audience includes engineers and scientists budgeting their model-training experiments, as well as non-practitioners trying to make sense of the economics of modern-day Natural Language Processing (NLP).

“SenseBERT: Driving Some Sense into BERT”
Original
Yoav Levine et al · 1908
Earlier work this paper cites.
“TinyBERT: Distilling BERT for Natural Language Understanding”, 2019
Original
Xiaoqi Jiao et al · 1909
Earlier work this paper cites.
“Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer”
Original
Colin Raffel et al · 1910
Earlier work this paper cites.
“Scaling Laws for Neural Language Models”, 2020
Original
Jared Kaplan et al · 2001
Earlier work this paper cites.
“Train Large, Then Compress: Rethinking Model Size for Efficient Training and Inference of Transformers”, 2020
Original
Zhuohan Li et al · 2002
Earlier work this paper cites.
“Visual Turing test for computer vision systems”
Donald Geman, Stuart Geman, Neil Hallonquist and Laurent Younes · 2015
Earlier work this paper cites.
“New Research From TSO Logic Shows AWS Costs Get Lower Every Year” Accessed: 2020-04-12, https://aws.amazon.com/blogs/apn/new-research-from-tso-logic-shows-aws-costs-get-lower-every-year/ , 2018
Aaron Rallo · 2018
Earlier work this paper cites.

“NLP’s ImageNet moment has arrived” Accessed: 2020-04-12, https://thegradient.pub/nlp-imagenet/ , 2018
Sebastian Ruder · 2018
Cited alongside, same era.
“DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter”
Victor Sanh, Lysandre Debut, Julien Chaumond and Thomas Wolf · 2019
Cited alongside, same era.
“BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding”
Jacob Devlin, Ming-Wei Chang, Kenton Lee and Kristina Toutanova · 2019
Cited alongside, same era.
“Show Your Work: Improved Reporting of Experimental Results”
Jesse Dodge et al · 2019
Cited alongside, same era.
“Language Models are Unsupervised Multitask Learners”, OpenAI’s Blog: https://d4mucfpksywv.cloudfront.net/better-language-models/language_models_are_unsupervised_multitask_learners.pdf , 2019
Alec Radford et al · 2019
Cited alongside, same era.
“Fixing the train-test resolution discrepancy”
Hugo Touvron, Andrea Vedaldi, Matthijs Douze and Herve Jegou · 2019
Cited alongside, same era.
“ResNet-50 Training Cost Comparison” Accessed: 2020-04-12, https://cloud.google.com/images/products/tpu/machine-learning-performance_2x.png , 2020
Google · 2020
Cited alongside, same era.

“National Research Cloud: Ensuring the Continuation of American Innovation” Accessed: 2020-04-12, https://hai.stanford.edu/news/national-research-cloud-ensuring-continuation-american-innovation , 2020
John Etchemendy and Fei-Fei Li · 2020
Closest in time.
“A Constructive Prediction of the Generalization Error Across Scales”
Jonathan. Rosenfeld, Amir Rosenfeld, Yonatan Belinkov and Nir Shavit · 2020
Closest in time.
“ELECTRA: Pre-training Text Encoders as Discriminators Rather Than Generators”
Kevin Clark, Minh-Thang Luong, Quoc. Le and Christopher. Manning · 2020
Closest in time.
“A New State of the Art in NLP: Beyond GPUs” Accessed: 2020-04-12, https://sambanova.ai/a-new-state-of-the-art-in-nlp-beyond-gpus/ , 2020
SambaNova · 2020
Closest in time.
“Reformer: The Efficient Transformer”
Nikita Kitaev, Lukasz Kaiser and Anselm Levskaya · 2020
Closest in time.
“ALBERT: A Lite BERT for Self-supervised Learning of Language Representations”
Zhenzhong Lan et al · 2020
Closest in time.

alphaXiv searches the wider corpus for related work and actual follow-ups.

alphaXiv is searching for related work…

Reading the bibliography…