Memory-efficient array redistribution through portable collective communication

Understand

Modern large-scale deep learning workloads highlight the need for parallel execution across many devices in order to fit model data into hardware accelerator memories.

• In these settings, array redistribution may be required during a computation, but can also become a bottleneck if not done efficiently.
• In this paper we address the problem of redistributing multi-dimensional array data in SPMD computations, the most prevalent form of parallelism in deep learning.
• We present a type-directed approach to synthesizing array redistributions as sequences of MPI-style collective operations.