Mime: Mimicking Centralized Stochastic Algorithms in Federated Learning

Understand

Federated learning (FL) is a challenging setting for optimization due to the heterogeneity of the data across different clients which gives rise to the client drift phenomenon.

• In fact, obtaining an algorithm for FL which is uniformly better than simple centralized training has been a major open problem thus far.
• In this work, we propose a general algorithmic framework, Mime, which i) mitigates client drift and ii) adapts arbitrary centralized optimization algorithms such as momentum and Adam to the cross-device federated learning setting.
• Mime uses a combination of control-variates and server-level statistics (e.g.