Efficient Communications in Training Large Scale Neural Networks

Understand

We consider the problem of how to reduce the cost of communication that is required for the parallel training of a neural network.

• The state-of-the-art method, Bulk Synchronous Parallel Stochastic Gradient Descent (BSP-SGD), requires many collective communication operations, like broadcasts of parameters or reductions for sub-gradient aggregations, which for large messages quickly dominates overall execution time and limits parallel scalability.
• To address this problem, we develop a new technique for collective operations, referred to as Linear Pipelining (LP).
• It is tuned to the message sizes that arise in BSP-SGD, and works effectively on multi-GPU systems.