Synthesizing Optimal Parallelism Placement and Reduction Strategies on Hierarchical Systems for Deep Learning

Understand

We present a novel characterization of the mapping of multiple parallelism forms (e.g.

• data and model parallelism) onto hierarchical accelerator systems that is hierarchy-aware and greatly reduces the space of software-to-hardware mapping.
• We experimentally verify the substantial effect of these mappings on all-reduce performance (up to 448x).
• We offer a novel syntax-guided program synthesis framework that is able to decompose reductions over one or more parallelism axes to sequences of collectives in a hierarchy- and mapping-aware way.