A Sharp Estimate on the Transient Time of Distributed Stochastic Gradient Descent

Understand

This paper is concerned with minimizing the average of $n$ cost functions over a network in which agents may communicate and exchange information with each other.

• We consider the setting where only noisy gradient information is available.
• To solve the problem, we study the distributed stochastic gradient descent (DSGD) method and perform a non-asymptotic convergence analysis.
• For strongly convex and smooth objective functions, DSGD asymptotically achieves the optimal network independent convergence rate compared to centralized stochastic gradient descent (SGD).