Error Feedback Fixes SignSGD and other Gradient Compression Schemes

Understand

Sign-based algorithms (e.g.

• signSGD) have been proposed as a biased gradient compression technique to alleviate the communication bottleneck in training large neural networks across multiple workers.
• We show simple convex counter-examples where signSGD does not converge to the optimum.
• Further, even when it does converge, signSGD may generalize poorly when compared with SGD.