Built on

Y. LeCun, C. Cortes, and C. Burges, “The MNIST database of handwritten digits,” http://yann.lecun.com/exdb/mnist/ , 1998
1998
Earlier work this paper cites.
D. Tse and P. Viswanath, Fundamentals of wireless communication . Cambridge, UK: Cambridge University Press, 2005
2005
Earlier work this paper cites.
D. L. Donoho, A. Maleki, and A. Montanari, “Message-passing algorithms for compressed sensing,” Proc. Nat. Acad. Sci. USA , vol. 106, no. 45, pp. 18 914–18 919, Nov. 2009
2009
Earlier work this paper cites.
S. Jakubczak and D. Katabi, “SoftCast: One-size-fits-all wireless video,” ACM SIGCOMM Comput. Commun. Rev. , vol. 40, no. 4, pp. 449–450, Oct. 2010
2010
Earlier work this paper cites.
Y. Polyanskiy, H. V. Poor, and S. Verdu, “Channel coding rate in the finite blocklength regime,” IEEE Trans. Inform. Theory , vol. 56, no. 5, pp. 2307–2359, May 2010
2010
Earlier work this paper cites.
F. Seide, H. Fu, J. Droppo, G. Li, and D. Yu, “1-bit stochastic gradient descent and its application to data-parallel distributed training of speech DNNs,” in INTERSPEECH , Singapore, Sep. 2014, pp. 1058–1062
2014
Earlier work this paper cites.
S. Gupta, A. Agrawal, K. Gopalakrishnan, and P. Narayanan, “Deep learning with limited numerical precision,” in ICML , Jul. 2015
2015
Earlier work this paper cites.
N. Strom, “Scalable distributed DNN training using commodity gpu cloud computing,” in INTERSPEECH , 2015
2015
Earlier work this paper cites.
J. Konecny and P. Richtarik, “Randomized distributed mean estimation: Accuracy vs communication,” arXiv:1611.07555 [cs.DC] , Nov. 2016
Original
2016
Earlier work this paper cites.
J. Konecny, H. B. McMahan, F. X. Yu, P. Richtarik, A. T. Suresh, and D. Bacon, “Federated learning: Strategies for improving communication efficiency,” arXiv:1610.05492v2 [cs.LG] , Oct. 2017
Original
2017
Earlier work this paper cites.
B. McMahan and D. Ramage, “Federated learning: Collaborative machine learning without centralized training data,” [online]. Available. https://ai.googleblog.com/2017/04/federated-learning-collaborative.html , Apr. 2017
2017
Earlier work this paper cites.
H. B. McMahan, E. Moore, D. Ramage, S. Hampson, and B. A. y Arcas, “Communication-efficient learning of deep networks from decentralized data,” in AISTATS , 2017
2017
Earlier work this paper cites.

Similar

V. Smith, C.-K. Chiang, M. Sanjabi, and A. S. Talwalkar, “Federated multi-task learning,” in Proc. Neural Information Processing Systems (NIPS) , Long Beach, CA, USA, 2017
2017
Cited alongside, same era.
D. Alistarh, D. Grubic, J. Z. Li, R. Tomioka, and M. Vojnovic, “QSGD: Communication-efficient SGD via randomized quantization and encoding,” in NIPS , Long Beach, CA, Dec. 2017, pp. 1709–1720
2017
Cited alongside, same era.
W. Wen, C. Xu, F. Yan, C. Wu, Y. Wang, Y. Chen, and H. Li, “TernGrad: Ternary gradients to reduce communication in distributed deep learning,” arXiv:1705.07878v6 [cs.LG] , Dec. 2017
Original
2017
Cited alongside, same era.
A. F. Aji and K. Heafield, “Sparse communication for distributed gradient descent,” arXiv:1704.05021v2 [cs.CL] , Jul. 2017
Original
2017
Cited alongside, same era.

Then

F. Sattler, S. Wiedemann, K. Muller, and W. Samek, “Sparse binary compression: Towards distributed deep learning with minimal communication,” arXiv:1805.08768v1 [cs.LG] , May 2018
Original
2018
Later among the works it cites.
C. Renggli, D. Alistarh, T. Hoefler, and M. Aghagolzadeh, “SparCML: High-performance sparse communication for machine learning,” arXiv:1802.08021v2 [cs.DC] , Oct. 2018
Original
2018
Later among the works it cites.
D. Alistarh, T. Hoefler, M. Johansson, S. Khirirat, N. Konstantinov, and C. Renggli, “The convergence of sparsified gradient methods,” arXiv:1809.10505v1 [cs.LG] , Sep. 2018
Original
2018
Later among the works it cites.
Y. Tsuzuku, H. Imachi, and T. Akiba, “Variance-based gradient compression for efficient distributed deep learning,” arXiv:1802.06058v2 [cs.LG] , Feb. 2018
Original
2018
Later among the works it cites.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…

2019

Federated Learning over Wireless Fading Channels

Amiri, Mohammad Mohammadi, Gunduz, Deniz

Understand

We study federated machine learning at the wireless network edge, where limited power wireless devices, each with its own dataset, build a joint model with the help of a remote parameter server (PS).

We consider a bandwidth-limited fading multiple access channel (MAC) from the wireless devices to the PS, and propose various techniques to implement distributed stochastic gradient descent (DSGD).
We first propose a digital DSGD (D-DSGD) scheme, in which one device is selected opportunistically for transmission at each iteration based on the channel conditions; the scheduled device quantizes its gradient estimate to a finite number of bits imposed by the channel condition, and transmits these bits to the PS in a reliable manner.
Next, motivated by the additive nature of the wireless MAC, we propose a novel analog communication scheme, referred to as the compressed analog DSGD (CA-DSGD), where the devices first sparsify their gradient estimates while accumulating error, and project the resultant sparse vector into a low-dimensional vector for bandwidth reduction.

Built on

Y. LeCun, C. Cortes, and C. Burges, “The MNIST database of handwritten digits,” http://yann.lecun.com/exdb/mnist/ , 1998
1998
Earlier work this paper cites.
D. Tse and P. Viswanath, Fundamentals of wireless communication . Cambridge, UK: Cambridge University Press, 2005
2005
Earlier work this paper cites.
D. L. Donoho, A. Maleki, and A. Montanari, “Message-passing algorithms for compressed sensing,” Proc. Nat. Acad. Sci. USA , vol. 106, no. 45, pp. 18 914–18 919, Nov. 2009
2009
Earlier work this paper cites.
S. Jakubczak and D. Katabi, “SoftCast: One-size-fits-all wireless video,” ACM SIGCOMM Comput. Commun. Rev. , vol. 40, no. 4, pp. 449–450, Oct. 2010
2010
Earlier work this paper cites.
Y. Polyanskiy, H. V. Poor, and S. Verdu, “Channel coding rate in the finite blocklength regime,” IEEE Trans. Inform. Theory , vol. 56, no. 5, pp. 2307–2359, May 2010
2010
Earlier work this paper cites.
F. Seide, H. Fu, J. Droppo, G. Li, and D. Yu, “1-bit stochastic gradient descent and its application to data-parallel distributed training of speech DNNs,” in INTERSPEECH , Singapore, Sep. 2014, pp. 1058–1062
2014
Earlier work this paper cites.
S. Gupta, A. Agrawal, K. Gopalakrishnan, and P. Narayanan, “Deep learning with limited numerical precision,” in ICML , Jul. 2015
2015
Earlier work this paper cites.
N. Strom, “Scalable distributed DNN training using commodity gpu cloud computing,” in INTERSPEECH , 2015
2015
Earlier work this paper cites.
J. Konecny and P. Richtarik, “Randomized distributed mean estimation: Accuracy vs communication,” arXiv:1611.07555 [cs.DC] , Nov. 2016
Original
2016
Earlier work this paper cites.
J. Konecny, H. B. McMahan, F. X. Yu, P. Richtarik, A. T. Suresh, and D. Bacon, “Federated learning: Strategies for improving communication efficiency,” arXiv:1610.05492v2 [cs.LG] , Oct. 2017
Original
2017
Earlier work this paper cites.
B. McMahan and D. Ramage, “Federated learning: Collaborative machine learning without centralized training data,” [online]. Available. https://ai.googleblog.com/2017/04/federated-learning-collaborative.html , Apr. 2017
2017
Earlier work this paper cites.
H. B. McMahan, E. Moore, D. Ramage, S. Hampson, and B. A. y Arcas, “Communication-efficient learning of deep networks from decentralized data,” in AISTATS , 2017
2017
Earlier work this paper cites.

Similar

V. Smith, C.-K. Chiang, M. Sanjabi, and A. S. Talwalkar, “Federated multi-task learning,” in Proc. Neural Information Processing Systems (NIPS) , Long Beach, CA, USA, 2017
2017
Cited alongside, same era.
D. Alistarh, D. Grubic, J. Z. Li, R. Tomioka, and M. Vojnovic, “QSGD: Communication-efficient SGD via randomized quantization and encoding,” in NIPS , Long Beach, CA, Dec. 2017, pp. 1709–1720
2017
Cited alongside, same era.
W. Wen, C. Xu, F. Yan, C. Wu, Y. Wang, Y. Chen, and H. Li, “TernGrad: Ternary gradients to reduce communication in distributed deep learning,” arXiv:1705.07878v6 [cs.LG] , Dec. 2017
Original
2017
Cited alongside, same era.
A. F. Aji and K. Heafield, “Sparse communication for distributed gradient descent,” arXiv:1704.05021v2 [cs.CL] , Jul. 2017
Original
2017
Cited alongside, same era.

Then

F. Sattler, S. Wiedemann, K. Muller, and W. Samek, “Sparse binary compression: Towards distributed deep learning with minimal communication,” arXiv:1805.08768v1 [cs.LG] , May 2018
Original
2018
Later among the works it cites.
C. Renggli, D. Alistarh, T. Hoefler, and M. Aghagolzadeh, “SparCML: High-performance sparse communication for machine learning,” arXiv:1802.08021v2 [cs.DC] , Oct. 2018
Original
2018
Later among the works it cites.
D. Alistarh, T. Hoefler, M. Johansson, S. Khirirat, N. Konstantinov, and C. Renggli, “The convergence of sparsified gradient methods,” arXiv:1809.10505v1 [cs.LG] , Sep. 2018
Original
2018
Later among the works it cites.
Y. Tsuzuku, H. Imachi, and T. Akiba, “Variance-based gradient compression for efficient distributed deep learning,” arXiv:1802.06058v2 [cs.LG] , Feb. 2018
Original
2018
Later among the works it cites.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

X. Sun, X. Ren, S. Ma, and H. Wang, “meProp: Sparsified back propagation for accelerated deep learning with reduced overfitting,” arXiv:1706.06197v4 [cs.LG] , Oct. 2017

Original

2017

Cited alongside, same era.

R. Xiong, J. Zhang, F. Wu, J. Xu, and W. Gao, “Power distortion optimization for uncoded linear transformed transmission of images and videos,” IEEE Trans. Image Process. , vol. 26, no. 1, pp. 222–236, Jan. 2017

2017

Cited alongside, same era.

D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” arXiv:1412.6980v9 [cs.LG] , Jan. 2017

Original

2017

Cited alongside, same era.

S. Zhou, Y. Wu, Z. Ni, X. Zhou, H. Wen, and Y. Zou, “DoReFa-Net: Training low bitwidth convolutional neural networks with low bitwidth gradients,” arXiv:1606.06160v3 [cs.NE] , Feb. 2018

Original

2018

Cited alongside, same era.

H. Wang, S. Sievert, S. Liu, Z. Charles, D. Papailiopoulos, and S. Wright, “ATOMO: Communication-efficient learning via atomic sparsification,” arXiv:1806.04090v2 [stat.ML] , Jun. 2018

Original

2018

Cited alongside, same era.

J. Bernstein, Y.-X. Wang, K. Azizzadenesheli, and A. Anandkumar, “signSGD: Compressed optimisation for non-convex problems,” arXiv:1802.04434v3 [cs.LG] , Aug. 2018

Original

2018

Cited alongside, same era.

B. Li, W. Wen, J. Mao, S. Li, Y. Chen, and H. Li, “Running sparse and low-precision neural network: When algorithm meets hardware,” in Proc. Asia and South Pacific Design Automation Conference (ASP-DAC) , Jeju, South Korea, Jan. 2018

2018

Cited alongside, same era.

Y. Lin, S. Han, H. Mao, Y. Wang, and W. J. Dally, “Deep gradient compression: Reducing the communication bandwidth for distributed training,” arXiv:1712.01887v2 [cs.CV] , Feb. 2018

Original

2018

Cited alongside, same era.

S. U. Stich, “Local SGD converges fast and communicates little,” arXiv:1805.09767v2 [math.OC] , Jun. 2018

Original

2018

Later among the works it cites.

T. Lin, S. U. Stich, and M. Jaggi, “Don’t use large mini-batches, use local SGD,” arXiv:1808.07217v3 [cs.LG] , Oct. 2018

Original

2018

Later among the works it cites.

T. Chen, G. B. Giannakis, T. Sun, and W. Yin, “LAG: Lazily aggregated gradient for communication-efficient distributed learning,” arXiv:1805.09965 [stat.ML] , May 2018

Original

2018

Later among the works it cites.

T-Y. Tung and D. Gündüz, “SparseCast: Hybrid digital-analog wireless image transmission exploiting frequency-domain sparsity,” IEEE Commun. Lett. , vol. 22, no. 12, pp. 2451–2454, Dec. 2018

2018

Later among the works it cites.

M. M. Amiri and D. Gündüz, “Machine learning at the wireless edge: Distributed stochastic gradient descent over-the-air,” arXiv:1901.00844 [cs.DC] , Jan. 2019

Original

2019

Closest in time.

G. Zhu, Y. Wang, and K. Huang, “Low-latency broadband analog aggregation for federated edge learning,” arXiv:1812.11494 [cs.IT] , Jan. 2019

Original

2019

Closest in time.

K. Yang, T. Jiang, Y. Shi, and Z. Ding, “Federated learning via over-the-air computation,” arXiv:1812.11750 [cs.LG] , Jan. 2019

Original

2019

Closest in time.

M. M. Amiri and D. Gündüz, “Over-the-air machine learning at the wireless edge,” in Proc. IEEE International Workshop on Signal Processing Advances in Wireless Communications (SPAWC) , Cannes, France, Jul. 2019, pp. 1–5

2019

Closest in time.