On Scale-out Deep Learning Training for Cloud and HPC

Understand

The exponential growth in use of large deep neural networks has accelerated the need for training these deep neural networks in hours or even minutes.

• This can only be achieved through scalable and efficient distributed training, since a single node/card cannot satisfy the compute, memory, and I/O requirements of today's state-of-the-art deep neural networks.
• However, scaling synchronous Stochastic Gradient Descent (SGD) is still a challenging problem and requires continued research/development.
• This entails innovations spanning algorithms, frameworks, communication libraries, and system design.