Error Compensated Distributed SGD Can Be Accelerated

Understand

Gradient compression is a recent and increasingly popular technique for reducing the communication cost in distributed training of large-scale machine learning models.

• In this work we focus on developing efficient distributed methods that can work for any compressor satisfying a certain contraction property, which includes both unbiased (after appropriate scaling) and biased compressors such as RandK and TopK.
• Applied naively, gradient compression introduces errors that either slow down convergence or lead to divergence.
• A popular technique designed to tackle this issue is error compensation/error feedback.