Taming Unbalanced Training Workloads in Deep Learning with Partial Collective Operations

Understand

Load imbalance pervasively exists in distributed deep learning training systems, either caused by the inherent imbalance in learned tasks or by the system itself.

• Traditional synchronous Stochastic Gradient Descent (SGD) achieves good accuracy for a wide variety of tasks, but relies on global synchronization to accumulate the gradients at every training step.
• In this paper, we propose eager-SGD, which relaxes the global synchronization for decentralized accumulation.
• To implement eager-SGD, we propose to use two partial collectives: solo and majority.