Gradient and Variable Tracking with Multiple Local SGD for Decentralized Non-Convex Learning

Understand

Stochastic distributed optimization methods that solve an optimization problem over a multi-agent network have played an important role in a variety of large-scale signal processing and machine leaning applications.

• Among the existing methods, the gradient tracking (GT) method is found robust against the variance between agents' local data distribution, in contrast to the distributed stochastic gradient descent (SGD) methods which have a slowed convergence speed when the agents have heterogeneous data distributions.
• However, the GT method can be communication expensive due to the need of a large number of iterations for convergence.
• In this paper, we intend to reduce the communication cost of the GT method by integrating it with the local SGD technique.