Memory-Efficient Pipeline-Parallel DNN Training

Understand

Many state-of-the-art ML results have been obtained by scaling up the number of parameters in existing models.

• However, parameters and activations for such large models often do not fit in the memory of a single accelerator device; this means that it is necessary to distribute training of large models over multiple accelerators.
• In this work, we propose PipeDream-2BW, a system that supports memory-efficient pipeline parallelism.
• PipeDream-2BW uses a novel pipelining and weight gradient coalescing strategy, combined with the double buffering of weights, to ensure high throughput, low memory footprint, and weight update semantics similar to data parallelism.