Exponential Graph is Provably Efficient for Decentralized Deep Training

Understand

Decentralized SGD is an emerging training method for deep learning known for its much less (thus faster) communication per iteration, which relaxes the averaging step in parallel SGD to inexact averaging.

• The less exact the averaging is, however, the more the total iterations the training needs to take.
• Therefore, the key to making decentralized SGD efficient is to realize nearly-exact averaging using little communication.
• This requires a skillful choice of communication topology, which is an under-studied topic in decentralized optimization.