Fetching the paper…
Reading the bibliography…
This paper investigates scaling laws for local SGD in LLM training, a distributed optimization algorithm that facilitates training on loosely connected devices.
Shaohuai Shi, Qiang Wang, Kaiyong Zhao, Zhenheng Tang, Yuxin Wang, Xiang Huang, and Xiaowen Chu · 1901
Earlier work this paper cites.
Zero: Memory optimizations toward training trillion parameter models, 2020, 1910.02054
Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, and Yuxiong He · 1910
Earlier work this paper cites.
Scaling laws for neural language models, 2020, 2001.08361
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei · 2001
Earlier work this paper cites.
Pytorch distributed: Experiences on accelerating data parallel training, 2020, 2006.15704
Shen Li, Yanli Zhao, Rohan Varma, Omkar Salpekar, Pieter Noordhuis, Teng Li, Adam Paszke, Jeff Smith, Brian Vaughan, Pritam Damania, and Soumith Chintala · 2006
Earlier work this paper cites.
Adam: A method for stochastic optimization, 2017, 1412.6980
Diederik P. Kingma and Jimmy Ba · 2017
Earlier work this paper cites.
Sgdr: Stochastic gradient descent with warm restarts, 2017, 1608.03983
Ilya Loshchilov and Frank Hutter · 2017
Earlier work this paper cites.
The convergence of sparsified gradient methods, 2018, 1809.10505
Dan Alistarh, Torsten Hoefler, Mikael Johansson, Sarit Khirirat, Nikola Konstantinov, and Cédric Renggli · 2018
Earlier work this paper cites.
Local sgd converges fast and communicates little, 2019, 1805.09767
Sebastian U. Stich · 2019
Earlier work this paper cites.
Exploring the limits of transfer learning with a unified text-to-text transformer
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu · 2020
Cited alongside, same era.
Efficient large-scale language model training on gpu clusters using megatron-lm, 2021, 2104.04473
Deepak Narayanan, Mohammad Shoeybi, Jared Casper, Patrick LeGresley, Mostofa Patwary, Vijay Anand Korthikanti, Dmitri Vainbrand, Prethvi Kashinkunti, Julie Bernauer, Bryan Catanzaro, Amar Phanishayee, and Matei Zaharia · 2021
Cited alongside, same era.
Hanlin Tang, Shaoduo Gan, Ammar Ahmad Awan, Samyam Rajbhandari, Conglong Li, Xiangru Lian, Ji Liu, Ce Zhang, and Yuxiong He · 2021
Cited alongside, same era.
Training compute-optimal large language models, 2022, 2203.15556
Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Tom Hennigan, Eric Noland, Katie Millican, George van den Driessche, Bogdan Damoc, Aurelia Guy, Simon Osindero, Karen Simonyan, Erich Elsen, Jack W. Rae, Oriol Vinyals, and Laurent Sifre · 2022
The llama 3 herd of models, 2024, 2407.21783
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, et al · 2024
Closest in time.
Alibaba hpn: A data center network for large language model training
Kun Qian, Yongqing Xi, Jiamin Cao, Jiaqi Gao, Yichi Xu, Yu Guan, Binzhang Fu, Xuemei Shi, Fangbo Zhu, Rui Miao, Chao Wang, Peng Wang, Pengcheng Zhang, Xianlong Zeng, Eddie Ruan, Zhiping Yao, Ennan Zhai, and Dennis Cai · 2024
Closest in time.
Samba: Simple hybrid state space models for efficient unlimited context language modeling
Liliang Ren, Yang Liu, Yadong Lu, Yelong Shen, Chen Liang, and Weizhu Chen · 2024
Closest in time.
Gemma 2: Improving open language models at a practical size, 2024, 2408.00118
Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupatiraju, et al · 2024
Closest in time.
alphaXiv searches the wider corpus for related work and actual follow-ups.
alphaXiv is searching for related work…
Cited alongside, same era.
Diloco: Distributed low-communication training of language models, 2023, 2311.08105
Arthur Douillard, Qixuan Feng, Andrei A. Rusu, Rachita Chhaparia, Yani Donchev, Adhiguna Kuncoro, Marc’Aurelio Ranzato, Arthur Szlam, and Jiajun Shen · 2023
Cited alongside, same era.
Communication-efficient learning of deep networks from decentralized data, 2023, 1602.05629
H. Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Agüera y Arcas · 2023
Cited alongside, same era.
SlimPajama: A 627B token cleaned and deduplicated version of RedPajama
Daria Soboleva, Faisal Al-Khateeb, Robert Myers, Jacob R Steeves, Joel Hestness, and Nolan Dey · 2023
Cited alongside, same era.
Deepseek llm: Scaling open-source language models with longtermism, 2024, 2401.02954
DeepSeek-AI, :, Xiao Bi, et al · 2024
Cited alongside, same era.
Weiyang Wang, Manya Ghobadi, Kayvon Shakeri, Ying Zhang, and Naader Hasani · 2024
Closest in time.
Qwen2 technical report, 2024, 2407.10671
An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, et al · 2024
Closest in time.
Chuxin: 1.6b technical report, 2024, 2405.04828
Xiaomin Zhuang, Yufan Jiang, Qiaozhi He, and Zhihua Wu · 2024
Closest in time.
Tinyllama: An open-source small language model
Peiyuan Zhang, Guangtao Zeng, Tianduo Wang, and Wei Lu · 2024
Closest in time.