A Comprehensive Performance Study of Large Language Models on Novel AI Accelerators

T. B. Brown, B. Mann, N. Ryder et al. , “Language models are few-shot learners,” 2020. [Online]. Available: https://arxiv.org/abs/2005.14165

Original

2005

Earlier work this paper cites.

A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is all you need,” Advances in neural information processing systems , vol. 30, 2017

2017

Earlier work this paper cites.

A. Mathuriya, D. Bard, P. Mendygral, L. Meadows, J. Arnemann, L. Shao, S. He, T. Kärnä, D. Moise, S. J. Pennycook, K. J. Maschhoff, J. Sewall, N. Kumar, S. Ho, M. F. Ringenburg, Prabhat, and V. W. Lee, “CosmoFlow: Using deep learning to learn the universe at scale,” in Proceedings of the International Conference for High Performance Computing, Networking, Storage, and Analysis, SC 2018, Dallas, TX, USA, November 11-16, 2018 . IEEE / ACM, 2018, pp. 65:1–65:11

2018

Earlier work this paper cites.

A. Radford, K. Narasimhan, T. Salimans, I. Sutskever et al. , “Improving language understanding by generative pre-training,” 2018. [Online]. Available: https://paperswithcode.com/paper/improving-language-understanding-by

2018

Earlier work this paper cites.

A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, I. Sutskever et al. , “Language models are unsupervised multitask learners,” OpenAI blog , vol. 1, no. 8, p. 9, 2019

2019

Earlier work this paper cites.

M. Shoeybi, M. Patwary, R. Puri, P. LeGresley, J. Casper, and B. Catanzaro, “Megatron-LM: Training multi-billion parameter language models using model parallelism,” ArXiv , vol. abs/1909.08053, 2019

Original

2019

Earlier work this paper cites.

A. Gokaslan and V. Cohen, “OpenWebText Corpus,” http://Skylion007.github.io/OpenWebTextCorpus , 2019

2019

Earlier work this paper cites.

R. Acciarri, C. Adams, C. Andreopoulos, J. Asaadi, M. Babicz, C. Backhouse, W. Badgett, L. Bagby, D. Barker, V. Basque et al. , “Cosmic background removal with deep neural networks in SBND,” arXiv preprint arXiv:2012.01301 , 2020

Original

2020

Earlier work this paper cites.

J. Jumper, R. Evans, A. Pritzel, T. Green, M. Figurnov, O. Ronneberger, K. Tunyasuvunakool, R. Bates, A. Žídek, A. Potapenko et al. , “Highly accurate protein structure prediction with AlphaFold,” Nature , vol. 596, no. 7873, pp. 583–589, 2021

2021

Earlier work this paper cites.

K. Tunyasuvunakool, J. Adler, Z. Wu, T. Green, M. Zielinski, A. Žídek, A. Bridgland, A. Cowie, C. Meyer, A. Laydon et al. , “Highly accurate protein structure prediction for the human proteome,” Nature , vol. 596, no. 7873, pp. 590–596, 2021

2021

Earlier work this paper cites.

J. Yin, A. Tsaris, S. Dash, R. Miller, F. Wang, and M. A. Shankar, “Comparative evaluation of deep learning workloads for leadership-class systems,” BenchCouncil Transactions on Benchmarks, Standards and Evaluations , vol. 1, no. 1, p. 100005, 2021. [Online]. Available: https://www.sciencedirect.com/science/article/pii/S2772485921000053

2021

Earlier work this paper cites.

D. Narayanan, M. Shoeybi, J. Casper, P. LeGresley, M. Patwary, V. Korthikanti, D. Vainbrand, P. Kashinkunti, J. Bernauer, B. Catanzaro, A. Phanishayee, and M. Zaharia, “Efficient large-scale language model training on GPU clusters using Megatron-LM,” in Proceedings of SC’21 , 2021

2021

Earlier work this paper cites.

M. Zvyagin, A. Brace, K. Hippe, Y. Deng, B. Zhang, C. O. Bohorquez, A. Clyde, B. Kale, D. Perez-Rivera, H. Ma et al. , “GenSLMs: Genome-scale language models reveal SARS-CoV-2 evolutionary dynamics,” bioRxiv , pp. 2022–10, 2022

2022

Earlier work this paper cites.

J. Pathak, S. Subramanian, P. Harrington, S. Raja, A. Chattopadhyay, M. Mardani, T. Kurth, D. Hall, Z. Li, K. Azizzadenesheli, P. Hassanzadeh, K. Kashinath, and A. Anandkumar, “FourCastNet: A global data-driven high-resolution weather model using adaptive Fourier neural operators,” 2022. [Online]. Available: https://arxiv.org/abs/2202.11214

Original

2022

Earlier work this paper cites.

T. L. Scao, A. Fan, C. Akiki, E. Pavlick, S. Ilić, D. Hesslow, R. Castagné, A. S. Luccioni, F. Yvon, M. Gallé et al. , “Bloom: A 176b-parameter open-access multilingual language model,” arXiv preprint arXiv:2211.05100 , 2022

Original

2022

Cited alongside, same era.

“LLM on Sambanova Datascale,” https://sambanova.ai/blog/achieving-best-in-class-large-language-model-accuracy-in-low-resource-settings/ , 2022

2022

Cited alongside, same era.

M. Emani, Z. Xie, S. Raskar, V. Sastry, W. Arnold, B. Wilson, R. Thakur, V. Vishwanath, Z. Liu, M. E. Papka et al. , “A comprehensive evaluation of novel AI accelerators for deep learning workloads,” in Proceedings of PMBS Workshop 2022 , 2022

2022

Cited alongside, same era.

“GenSLMs: Genome-scale language models,” https://github.com/ramanathanlab/genslm , 2022

2022

Cited alongside, same era.

“Habana Gaudi2- Memory Efficient training,” https://developer.habana.ai/blog/memory-efficient-training-on-habana-gaudi-with-deepspeed/ , 2023

2023

Closest in time.

V. Srinivasan, D. Gandhi, U. Thakker, and R. Prabhakar, “Training large language models efficiently with sparsity and dataflow,” 2023. [Online]. Available: https://arxiv.org/abs/2304.05511

Original

2023

Closest in time.

V. Thangarasa, A. Gupta, W. Marshall, T. Li, K. Leong, D. DeCoste, S. Lie, and S. Saxena, “SPDF: Sparse pre-training and dense fine-tuning for large language models,” arXiv preprint arXiv:2303.10464 , 2023

Original

2023

Closest in time.

S. Saxena, V. Thangarasa, A. Gupta, and S. Lie, “Sparse iso-flop transformations for maximizing training efficiency,” 2023. [Online]. Available: https://arxiv.org/abs/2303.11525

Original

2023

Closest in time.