Periodic Stochastic Gradient Descent with Momentum for Decentralized Training

Understand

Decentralized training has been actively studied in recent years.

• Although a wide variety of methods have been proposed, yet the decentralized momentum SGD method is still underexplored.
• In this paper, we propose a novel periodic decentralized momentum SGD method, which employs the momentum schema and periodic communication for decentralized training.
• With these two strategies, as well as the topology of the decentralized training system, the theoretical convergence analysis of our proposed method is difficult.