Multi-Level Local SGD for Heterogeneous Hierarchical Networks

Understand

We propose Multi-Level Local SGD, a distributed gradient method for learning a smooth, non-convex objective in a heterogeneous multi-level network.

• Our network model consists of a set of disjoint sub-networks, with a single hub and multiple worker nodes; further, worker nodes may have different operating rates.
• The hubs exchange information with one another via a connected, but not necessarily complete communication network.
• In our algorithm, sub-networks execute a distributed SGD algorithm, using a hub-and-spoke paradigm, and the hubs periodically average their models with neighboring hubs.