Parallel Restarted SGD with Faster Convergence and Less Communication: Demystifying Why Model Averaging Works for Deep Learning

Understand

In distributed training of deep neural networks, parallel mini-batch SGD is widely used to speed up the training process by using multiple workers.

• It uses multiple workers to sample local stochastic gradient in parallel, aggregates all gradients in a single server to obtain the average, and update each worker's local model using a SGD update with the averaged gradient.
• Ideally, parallel mini-batch SGD can achieve a linear speed-up of the training time (with respect to the number of workers) compared with SGD over a single worker.
• However, such linear scalability in practice is significantly limited by the growing demand for gradient communication as more workers are involved.