Quantized Decentralized Stochastic Learning over Directed Graphs

Understand

We consider a decentralized stochastic learning problem where data points are distributed among computing nodes communicating over a directed graph.

• As the model size gets large, decentralized learning faces a major bottleneck that is the heavy communication load due to each node transmitting large messages (model updates) to its neighbors.
• To tackle this bottleneck, we propose the quantized decentralized stochastic learning algorithm over directed graphs that is based on the push-sum algorithm in decentralized consensus optimization.
• More importantly, we prove that our algorithm achieves the same convergence rates of the decentralized stochastic learning algorithm with exact-communication for both convex and non-convex losses.