Alpa: Automating Inter- and Intra-Operator Parallelism for Distributed Deep Learning

Understand

Alpa automates model-parallel training of large deep learning (DL) models by generating execution plans that unify data, operator, and pipeline parallelism.

• Existing model-parallel training systems either require users to manually create a parallelization plan or automatically generate one from a limited space of model parallelism configurations.
• They do not suffice to scale out complex DL models on distributed compute devices.
• Alpa distributes the training of large DL models by viewing parallelisms as two hierarchical levels: inter-operator and intra-operator parallelisms.