Effective Federated Adaptive Gradient Methods with Non-IID Decentralized Data

Understand

Federated learning allows loads of edge computing devices to collaboratively learn a global model without data sharing.

• The analysis with partial device participation under non-IID and unbalanced data reflects more reality.
• In this work, we propose federated learning versions of adaptive gradient methods - Federated AGMs - which employ both the first-order and second-order momenta, to alleviate generalization performance deterioration caused by dissimilarity of data population among devices.
• To further improve the test performance, we compare several schemes of calibration for the adaptive learning rate, including the standard Adam calibrated by $\epsilon$, $p$-Adam, and one calibrated by an activation function.