AdaComp : Adaptive Residual Gradient Compression for Data-Parallel Distributed Training

Understand

Highly distributed training of Deep Neural Networks (DNNs) on future compute platforms (offering 100 of TeraOps/s of computational capacity) is expected to be severely communication constrained.

• To overcome this limitation, new gradient compression techniques are needed that are computationally friendly, applicable to a wide variety of layers seen in Deep Neural Networks and adaptable to variations in network architectures as well as their hyper-parameters.
• In this paper we introduce a novel technique - the Adaptive Residual Gradient Compression (AdaComp) scheme.
• AdaComp is based on localized selection of gradient residues and automatically tunes the compression rate depending on local activity.