Enhancing Stability for Large Language Models Training in Constrained Bandwidth Networks

Crafting papers on machine learning

Langley, P · 2000

Earlier work this paper cites.

Horovod: fast and easy distributed deep learning in TensorFlow

Original

Sergeev, A. and Balso, M. D · 2018

Earlier work this paper cites.

Evaluating modern gpu interconnect: Pcie, nvlink, nv-sli, nvswitch and gpudirect, 2019

Li, A., Song, S. L., Chen, J., Li, J., Liu, X., Tallent, N., and Barker, K · 2019

Earlier work this paper cites.

Performance analysis of deep learning workloads on leading-edge systems, 2019

Ren, Y., Yoo, S., and Hoisie, A · 2019

Earlier work this paper cites.

Language models are few-shot learners

Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al · 2020

Earlier work this paper cites.

Measuring massive multitask language understanding

Original

Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D., and Steinhardt, J · 2020

Earlier work this paper cites.

A unified architecture for accelerating distributed DNN training in heterogeneous GPU/CPU clusters

Jiang, Y., Zhu, Y., Lan, C., Yi, B., Cui, Y., and Guo, C · 2020

Earlier work this paper cites.

Pytorch distributed: experiences on accelerating data parallel training

Li, S., Zhao, Y., Varma, R., Salpekar, O., Noordhuis, P., Li, T., Paszke, A., Smith, J., Vaughan, B., Damania, P., and Chintala, S · 2020

Cited alongside, same era.

Zero: Memory optimizations toward training trillion parameter models

Rajbhandari, S., Rasley, J., Ruwase, O., and He, Y · 2020

Cited alongside, same era.

Megatron-lm: Training multi-billion parameter language models using model parallelism, 2020

Shoeybi, M., Patwary, M., Puri, R., LeGresley, P., Casper, J., and Catanzaro, B · 2020

Cited alongside, same era.

{ \{ Zero-offload } \} : Democratizing { \{ billion-scale } \} model training

Ren, J., Rajbhandari, S., Aminabadi, R. Y., Ruwase, O., Yang, S., Zhang, M., Li, D., and He, Y · 2021

Cited alongside, same era.

Using deepspeed and megatron to train megatron-turing nlg 530b, a large-scale generative language model

Original

Smith, S., Patwary, M., Norick, B., LeGresley, P., Rajbhandari, S., Casper, J., Liu, Z., Prabhumoye, S., Zerveas, G., Korthikanti, V., et al · 2022

The falcon series of open language models, 2023

Almazrouei, E., Alobeidli, H., Alshamsi, A., Cappelli, A., Cojocaru, R., Debbah, M., Étienne Goffinet, Hesslow, D., Launay, J., Malartic, Q., Mazzotta, D., Noune, B., Pannier, B., and Penedo, G · 2023

Later among the works it cites.

Mistral 7b

Original

Jiang, A. Q., Sablayrolles, A., Mensch, A., Bamford, C., Chaplot, D. S., Casas, D. d. l., Bressand, F., Lengyel, G., Lample, G., Saulnier, L., et al · 2023

Later among the works it cites.

Llama 2: Open foundation and fine-tuned chat models

Original

Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., et al · 2023

Later among the works it cites.

Zero++: Extremely efficient collective communication for giant model training

Original

Wang, G., Qin, H., Jacobs, S. A., Holmes, C., Rajbhandari, S., Ruwase, O., Yan, F., Yang, L., and He, Y · 2023

Later among the works it cites.