Pipeline Parallelism with Controllable Memory

Training deep nets with sublinear memory cost

Original

Tianqi Chen, Bing Xu, Chiyuan Zhang, and Carlos Guestrin · 2016

Earlier work this paper cites.

Ampnet: Asynchronous model-parallel training for dynamic neural networks

Original

Alexander L Gaunt, Matthew A Johnson, Maik Riechert, Daniel Tarlow, Ryota Tomioka, Dimitrios Vytiniotis, and Sam Webster · 2017

Earlier work this paper cites.

Accurate, large minibatch sgd: Training imagenet in 1 hour

Original

Priya Goyal, Piotr Dollár, Ross Girshick, Pieter Noordhuis, Lukasz Wesolowski, Aapo Kyrola, Andrew Tulloch, Yangqing Jia, and Kaiming He · 2017

Earlier work this paper cites.

Pipedream: Fast and efficient pipeline parallel dnn training

Original

Aaron Harlap, Deepak Narayanan, Amar Phanishayee, Vivek Seshadri, Nikhil Devanur, Greg Ganger, and Phil Gibbons · 2018

Earlier work this paper cites.

Asynchronous decentralized parallel stochastic gradient descent

Xiangru Lian, Wei Zhang, Ce Zhang, and Ji Liu · 2018

Earlier work this paper cites.

Gpipe: Efficient training of giant neural networks using pipeline parallelism

Yanping Huang, Youlong Cheng, Ankur Bapna, Orhan Firat, Dehao Chen, Mia Chen, HyoukJoong Lee, Jiquan Ngiam, Quoc V Le, Yonghui Wu, et al · 2019

Earlier work this paper cites.

Megatron-lm: Training multi-billion parameter language models using model parallelism

Original

Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro · 2019

Earlier work this paper cites.

Language models are few-shot learners

Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al · 2020

Earlier work this paper cites.

Gems: Gpu-enabled memory-aware model-parallelism system for distributed dnn training

Arpan Jain, Ammar Ahmad Awan, Asmaa M Aljuhani, Jahanzeb Maqbool Hashmi, Quentin G Anthony, Hari Subramoni, Dhableswar K Panda, Raghu Machiraju, and Anil Parwani · 2020

Cited alongside, same era.

Communication-efficient distributed deep learning: A comprehensive survey

Original

Zhenheng Tang, Shaohuai Shi, Wei Wang, Bo Li, and Xiaowen Chu · 2020

Cited alongside, same era.

Dapple: A pipelined data parallel approach for training large models

Shiqing Fan, Yi Rong, Chen Meng, Zongyan Cao, Siyu Wang, Zhen Zheng, Chuan Wu, Guoping Long, Jun Yang, Lixue Xia, et al · 2021

Cited alongside, same era.

Chimera: efficiently training large-scale neural networks with bidirectional pipelines

Shigang Li and Torsten Hoefler · 2021

Cited alongside, same era.

Efficient large-scale language model training on gpu clusters using megatron-lm

Deepak Narayanan, Mohammad Shoeybi, Jared Casper, Patrick LeGresley, Mostofa Patwary, Vijay Korthikanti, Dmitri Vainbrand, Prethvi Kashinkunti, Julie Bernauer, Bryan Catanzaro, et al · 2021

Alpa: Automating inter-and { \{ Intra-Operator } \} parallelism for distributed deep learning

Lianmin Zheng, Zhuohan Li, Hao Zhang, Yonghao Zhuang, Zhifeng Chen, Yanping Huang, Yida Wang, Yuanzhong Xu, Danyang Zhuo, Eric P Xing, et al · 2022

Later among the works it cites.

Flashattention-2: Faster attention with better parallelism and work partitioning

Original

Tri Dao · 2023

Later among the works it cites.

Bpipe: Memory-balanced pipeline parallelism for training large language models

Taebum Kim, Hyoungjoo Kim, Gyeong-In Yu, and Byung-Gon Chun · 2023

Later among the works it cites.

Reducing activation recomputation in large transformer models

Vijay Anand Korthikanti, Jared Casper, Sangkug Lym, Lawrence McAfee, Michael Andersch, Mohammad Shoeybi, and Bryan Catanzaro · 2023

Later among the works it cites.

Hanayo: Harnessing wave-like pipeline parallelism for enhanced large model training efficiency

Ziming Liu, Shenggan Cheng, Hao Zhou, and Yang You · 2023

Later among the works it cites.