Zero Bubble Pipeline Parallelism

Understand

Pipeline parallelism is one of the key components for large-scale distributed training, yet its efficiency suffers from pipeline bubbles which were deemed inevitable.

• In this work, we introduce a scheduling strategy that, to our knowledge, is the first to successfully achieve zero pipeline bubbles under synchronous training semantics.
• The key idea behind this improvement is to split the backward computation into two parts, one that computes gradient for the input and another that computes for the parameters.
• Based on this idea, we handcraft novel pipeline schedules that significantly outperform the baseline methods.