Revisiting Distributed Synchronous SGD

Understand

Distributed training of deep learning models on large-scale training data is typically conducted with asynchronous stochastic optimization to maximize the rate of updates, at the cost of additional noise introduced from asynchrony.

• In contrast, the synchronous approach is often thought to be impractical due to idle time wasted on waiting for straggling workers.
• We revisit these conventional beliefs in this paper, and examine the weaknesses of both approaches.
• We demonstrate that a third approach, synchronous optimization with backup workers, can avoid asynchronous noise while mitigating for the worst stragglers.