Fetching the paper…

Reading the bibliography…

2020

Communication Efficient Federated Learning over Multiple Access Channels

Chang, Wei-Ting, Tandon, Ravi

Original
PDF
Post on X

Understand

In this work, we study the problem of federated learning (FL), where distributed users aim to jointly train a machine learning model with the help of a parameter server (PS).

In each iteration of FL, users compute local gradients, followed by transmission of the quantized gradients for subsequent aggregation and model updates at PS.
One of the challenges of FL is that of communication overhead due to FL's iterative nature and large model sizes.
One recent direction to alleviate communication bottleneck in FL is to let users communicate simultaneously over a multiple access channel (MAC), possibly making better use of the communication resources.

Communication Efficient Federated Learning over Multiple Access Channels · Around

Built on

M. M. Amiri and D. Gündüz, “Machine learning at the wireless edge: Distributed stochastic gradient descent over-the-air,” CoRR , vol. abs/1901.00844, 2019. [Online]. Available: http://arxiv.org/abs/1901.00844
Original
1901
Earlier work this paper cites.
M. M. Amiri and D. Gündüz, “Federated learning over wireless fading channels,” CoRR , vol. abs/1907.09769, 2019. [Online]. Available: http://arxiv.org/abs/1907.09769
Original
1907
Earlier work this paper cites.
M. M. Amiri, T. M. Duman, and D. Gündüz, “Collaborative machine learning at the wireless edge with blind transmitters,” CoRR , vol. abs/1907.03909, 2019. [Online]. Available: http://arxiv.org/abs/1907.03909
Original
1907
Earlier work this paper cites.
Q. Zeng, Y. Du, K. K. Leung, and K. Huang, “Energy-efficient radio resource allocation for federated edge learning,” CoRR , vol. abs/1907.06040, 2019. [Online]. Available: http://arxiv.org/abs/1907.06040
Original
1907
Earlier work this paper cites.
T. Sery and K. Cohen, “On analog gradient descent learning over multiple access fading channels,” CoRR , vol. abs/1908.07463, 2019. [Online]. Available: http://arxiv.org/abs/1908.07463
Original
1908
Earlier work this paper cites.
M. S. H. Abad, E. Ozfatura, D. Gündüz, and O. Ercetin, “Hierarchical federated learning across heterogeneous cellular networks,” CoRR , vol. abs/1909.02362, 2019. [Online]. Available: http://arxiv.org/abs/1909.02362
Original
1909
Earlier work this paper cites.
M. Chen, Z. Yang, W. Saad, C. Yin, H. V. Poor, and S. Cui, “A joint learning and communications framework for federated learning over wireless networks,” CoRR , vol. abs/1909.07972, 2019. [Online]. Available: http://arxiv.org/abs/1909.07972
Original
1909
Earlier work this paper cites.
Y. Sun, S. Zhou, and D. Gündüz, “Energy-aware analog aggregation for federated learning with redundant data,” CoRR , vol. abs/1911.00188, 2019. [Online]. Available: http://arxiv.org/abs/1911.00188
Original
1911
Earlier work this paper cites.
A. Schrijver, Theory of linear and integer programming . John Wiley & Sons, 1998
1998
Earlier work this paper cites.

Similar

T. M. Cover and J. A. Thomas, Elements of Information Theory . Wiley-Interscience, 2006
2006
Cited alongside, same era.
A. Rakhlin, O. Shamir, and K. Sridharan, “Making gradient descent optimal for strongly convex stochastic optimization,” CoRR , vol. abs/1109.5647, 2012. [Online]. Available: http://arxiv.org/abs/1109.5647
Original
2012
Cited alongside, same era.
F. Seide, H. Fu, J. Droppo, G. Li, and D. Yu, “1-Bit Stochastic Gradient Descent and Application to Data-Parallel Distributed Training of Speech DNNs,” in Interspeech 2014 , September 2014
2014
Cited alongside, same era.
N. Dryden, S. A. Jacobs, T. Moon, and B. Van Essen, “Communication quantization for data-parallel training of deep neural networks,” in Proceedings of the Workshop on Machine Learning in High Performance Computing Environments , ser. MLHPC ’16. Piscataway, NJ, USA: IEEE Press, 2016, pp. 1–8. [Online]. Available: https://doi.org/10.1109/MLHPC.2016.4
2016
Cited alongside, same era.

Then

J. Bernstein, Y.-X. Wang, K. Azizzadenesheli, and A. Anandkumar, “signSGD: Compressed optimisation for non-convex problems,” in Proceedings of the 35th International Conference on Machine Learning , ser. Proceedings of Machine Learning Research, vol. 80, 10–15 Jul 2018, pp. 560–569
2018
Later among the works it cites.
J. Wangni, J. Wang, J. Liu, and T. Zhang, “Gradient sparsification for communication-efficient distributed optimization,” in Advances in Neural Information Processing Systems 31 , 2018, pp. 1299–1309
2018
Later among the works it cites.
F. Sattler, S. Wiedemann, K. Müller, and W. Samek, “Sparse binary compression: Towards distributed deep learning with minimal communication,” CoRR , vol. abs/1805.08768, 2018. [Online]. Available: http://arxiv.org/abs/1805.08768
Original
2018
Later among the works it cites.
K. Yang, T. Jiang, Y. Shi, and Z. Ding, “Federated learning via over-the-air computation,” CoRR , vol. abs/1812.11750, 2018. [Online]. Available: http://arxiv.org/abs/1812.11750
Original
2018

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

W. Wen, C. Xu, F. Yan, C. Wu, Y. Wang, Y. Chen, and H. Li, “TernGrad: Ternary Gradients to Reduce Communication in Distributed Deep Learning,” in Advances in Neural Information Processing Systems 30 , 2017, pp. 1509–1519

2017

Cited alongside, same era.

A. F. Aji and K. Heafield, “Sparse communication for distributed gradient descent,” CoRR , vol. abs/1704.05021, 2017. [Online]. Available: http://arxiv.org/abs/1704.05021

2017

Cited alongside, same era.

Y. Lin, S. Han, H. Mao, Y. Wang, and W. J. Dally, “Deep gradient compression: Reducing the communication bandwidth for distributed training,” CoRR , vol. abs/1712.01887, 2017. [Online]. Available: http://arxiv.org/abs/1712.01887

Original

2017

Cited alongside, same era.

D. Alistarh, D. Grubic, J. Li, R. Tomioka, and M. Vojnovic, “QSGD: Communication-Efficient SGD via Gradient Quantization and Encoding,” in Advances in Neural Information Processing Systems 30 , 2017, pp. 1709–1720

2017

Cited alongside, same era.

A. T. Suresh, F. X. Yu, S. Kumar, and H. B. McMahan, “Distributed mean estimation with limited communication,” in Proceedings of the 34th International Conference on Machine Learning , 2017, p. 3329–3337

2017

Cited alongside, same era.

Later among the works it cites.

S. Wang, T. Tuor, T. Salonidis, K. K. Leung, C. Makaya, T. He, and K. Chan, “When edge meets learning: Adaptive control for resource-constrained distributed machine learning,” CoRR , vol. abs/1804.05271, 2018. [Online]. Available: http://arxiv.org/abs/1804.05271

2018

Later among the works it cites.

N. Agarwal, A. T. Suresh, F. Yu, S. Kumar, and H. B. Mcmahan, “cpSGD: Communication-efficient and differentially-private distributed SGD,” CoRR , vol. abs/1805.10559, 2018. [Online]. Available: http://arxiv.org/abs/1805.10559

2018

Later among the works it cites.

M. M. Amiri and D. Gündüz, “Over-the-air machine learning at the wireless edge,” in 2019 IEEE 20th International Workshop on Signal Processing Advances in Wireless Communications (SPAWC) , July 2019, pp. 1–5

2019

Later among the works it cites.

T. Sery and K. Cohen, “A sequential gradient-based multiple access for distributed learning over fading channels,” in 2019 57th Annual Allerton Conference on Communication, Control, and Computing (Allerton) , Sep. 2019, pp. 303–307

2019

Later among the works it cites.

G. Zhu, Y. Wang, and K. Huang, “Broadband analog aggregation for low-latency federated edge learning,” IEEE Transactions on Wireless Communications , vol. 19, no. 1, pp. 491–506, Jan. 2020

2020

Closest in time.