Built on

C. Ma, V. Smith, M. Jaggi, M.I. Jordan, P. Richtárik, and M. Takáč, Adding vs. averaging in distributed primal-dual optimization , in The 32nd International Conference on Machine Learning . 2015, pp. 1973–1982
1982
Earlier work this paper cites.
O. Fercoq, Z. Qu, P. Richtárik, and M. Takáč, Fast distributed coordinate descent for minimizing non-strongly convex losses , IEEE International Workshop on Machine Learning for Signal Processing (2014)
2014
Earlier work this paper cites.
M. Jaggi, V. Smith, M. Takáč, J. Terhorst, S. Krishnan, T. Hofmann, and M.I. Jordan, Communication-efficient distributed dual coordinate ascent , in Advances in Neural Information Processing Systems 27 . 2014
2014
Earlier work this paper cites.
F. Seide, H. Fu, J. Droppo, G. Li, and D. Yu, 1-bit stochastic gradient descent and its application to data-parallel distributed training of speech dnns , in Fifteenth Annual Conference of the International Speech Communication Association . 2014
2014
Earlier work this paper cites.
O. Shamir, N. Srebro, and T. Zhang, Communication-Efficient Distributed Optimization using an Approximate Newton-type Method , in Proceedings of the 31st International Conference on Machine Learning, PMLR , Vol. 32. 2014, pp. 1000–1008
2014
Earlier work this paper cites.
C. Ma and M. Takáč, Partitioning data on features or samples in communication-efficient distributed optimization? , OptML@NIPS 2015, arXiv:1510.06688 (2015)
Original
2015
Earlier work this paper cites.
Y. Zhang and L. Xiao, DiSCO: Distributed Optimization for Self-Concordant Empirical Loss , in Proceedings of the 32nd International Conference on Machine Learning, PMLR , Vol. 37. 2015, pp. 362–370
2015
Earlier work this paper cites.
J. Konečný, H.B. McMahan, F.X. Yu, P. Richtárik, A.T. Suresh, and D. Bacon, Federated learning: Strategies for improving communication efficiency , arXiv preprint arXiv:1610.05492 (2016)
Original
2016
Earlier work this paper cites.
J. Konečný and P. Richtárik, Randomized distributed mean estimation: Accuracy vs communication , arXiv preprint arXiv:1611.07555 (2016)
Original
2016
Earlier work this paper cites.
S.J. Reddi, J. Konečný, P. Richtárik, B. Póczós, and A. Smola, AIDE: Fast and communication efficient distributed optimization , arXiv preprint arXiv:1608.06879 (2016)
Original
2016
Earlier work this paper cites.

Similar

P. Richtárik and M. Takáč, Distributed coordinate descent method for learning with big data , Journal of Machine Learning Research 17 (2016), pp. 1–25
2016
Cited alongside, same era.
D. Alistarh, D. Grubic, J. Li, R. Tomioka, and M. Vojnovic, QSGD: Communication-efficient SGD via gradient quantization and encoding , in Advances in Neural Information Processing Systems . 2017, pp. 1709–1720
2017
Cited alongside, same era.
S. Chunduri, P. Coffman, S. Parker, and K. Kumaran, Performance analysis of mpi on cray xc40 xeon phi system (2017)
2017
Cited alongside, same era.
C. Ma, M. Jaggi, F.E. Curtis, N. Srebro, and M. Takáč, An accelerated communication-efficient primal-dual optimization framework for structured machine learning , arXiv:1711.05305 (2017)
Original
2017
Cited alongside, same era.

Then

S. Horváth, D. Kovalev, K. Mishchenko, S. Stich, and P. Richtárik, Stochastic distributed learning with gradient quantization and variance reduction , arXiv preprint arXiv:1904.05115 (2019)
Original
2019
Closest in time.
K. Mishchenko, E. Gorbunov, M. Takáč, and P. Richtárik, Distributed learning with compressed gradient differences , arXiv preprint arXiv:1901.09269 (2019)
Original
2019
Closest in time.
E. Gorbunov, D. Kovalev, D. Makarenko, and P. Richtárik, Linearly converging error compensated sgd , Advances in Neural Information Processing Systems 33 (2020), pp. 20889–20900
2020
Closest in time.
Z. Li, D. Kovalev, X. Qian, and P. Richtárik, Acceleration for compressed gradient descent in distributed and federated optimization , arXiv preprint arXiv:2002.11364 (2020)
Original
2020
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…

2019

Distributed Learning with Compressed Gradient Differences

Mishchenko, Konstantin, Gorbunov, Eduard, Takáč, Martin et al.

Understand

Training large machine learning models requires a distributed computing approach, with communication of the model updates being the bottleneck.

For this reason, several methods based on the compression (e.g., sparsification and/or quantization) of updates were recently proposed, including QSGD (Alistarh et al., 2017), TernGrad (Wen et al., 2017), SignSGD (Bernstein et al., 2018), and DQGD (Khirirat et al., 2018).
However, none of these methods are able to learn the gradients, which renders them incapable of converging to the true optimum in the batch mode.
In this work we propose a new distributed learning method -- DIANA -- which resolves this issue via compression of gradient differences.

Built on

C. Ma, V. Smith, M. Jaggi, M.I. Jordan, P. Richtárik, and M. Takáč, Adding vs. averaging in distributed primal-dual optimization , in The 32nd International Conference on Machine Learning . 2015, pp. 1973–1982
1982
Earlier work this paper cites.
O. Fercoq, Z. Qu, P. Richtárik, and M. Takáč, Fast distributed coordinate descent for minimizing non-strongly convex losses , IEEE International Workshop on Machine Learning for Signal Processing (2014)
2014
Earlier work this paper cites.
M. Jaggi, V. Smith, M. Takáč, J. Terhorst, S. Krishnan, T. Hofmann, and M.I. Jordan, Communication-efficient distributed dual coordinate ascent , in Advances in Neural Information Processing Systems 27 . 2014
2014
Earlier work this paper cites.
F. Seide, H. Fu, J. Droppo, G. Li, and D. Yu, 1-bit stochastic gradient descent and its application to data-parallel distributed training of speech dnns , in Fifteenth Annual Conference of the International Speech Communication Association . 2014
2014
Earlier work this paper cites.
O. Shamir, N. Srebro, and T. Zhang, Communication-Efficient Distributed Optimization using an Approximate Newton-type Method , in Proceedings of the 31st International Conference on Machine Learning, PMLR , Vol. 32. 2014, pp. 1000–1008
2014
Earlier work this paper cites.
C. Ma and M. Takáč, Partitioning data on features or samples in communication-efficient distributed optimization? , OptML@NIPS 2015, arXiv:1510.06688 (2015)
Original
2015
Earlier work this paper cites.
Y. Zhang and L. Xiao, DiSCO: Distributed Optimization for Self-Concordant Empirical Loss , in Proceedings of the 32nd International Conference on Machine Learning, PMLR , Vol. 37. 2015, pp. 362–370
2015
Earlier work this paper cites.
J. Konečný, H.B. McMahan, F.X. Yu, P. Richtárik, A.T. Suresh, and D. Bacon, Federated learning: Strategies for improving communication efficiency , arXiv preprint arXiv:1610.05492 (2016)
Original
2016
Earlier work this paper cites.
J. Konečný and P. Richtárik, Randomized distributed mean estimation: Accuracy vs communication , arXiv preprint arXiv:1611.07555 (2016)
Original
2016
Earlier work this paper cites.
S.J. Reddi, J. Konečný, P. Richtárik, B. Póczós, and A. Smola, AIDE: Fast and communication efficient distributed optimization , arXiv preprint arXiv:1608.06879 (2016)
Original
2016
Earlier work this paper cites.

Similar

P. Richtárik and M. Takáč, Distributed coordinate descent method for learning with big data , Journal of Machine Learning Research 17 (2016), pp. 1–25
2016
Cited alongside, same era.
D. Alistarh, D. Grubic, J. Li, R. Tomioka, and M. Vojnovic, QSGD: Communication-efficient SGD via gradient quantization and encoding , in Advances in Neural Information Processing Systems . 2017, pp. 1709–1720
2017
Cited alongside, same era.
S. Chunduri, P. Coffman, S. Parker, and K. Kumaran, Performance analysis of mpi on cray xc40 xeon phi system (2017)
2017
Cited alongside, same era.
C. Ma, M. Jaggi, F.E. Curtis, N. Srebro, and M. Takáč, An accelerated communication-efficient primal-dual optimization framework for structured machine learning , arXiv:1711.05305 (2017)
Original
2017
Cited alongside, same era.

Then

S. Horváth, D. Kovalev, K. Mishchenko, S. Stich, and P. Richtárik, Stochastic distributed learning with gradient quantization and variance reduction , arXiv preprint arXiv:1904.05115 (2019)
Original
2019
Closest in time.
K. Mishchenko, E. Gorbunov, M. Takáč, and P. Richtárik, Distributed learning with compressed gradient differences , arXiv preprint arXiv:1901.09269 (2019)
Original
2019
Closest in time.
E. Gorbunov, D. Kovalev, D. Makarenko, and P. Richtárik, Linearly converging error compensated sgd , Advances in Neural Information Processing Systems 33 (2020), pp. 20889–20900
2020
Closest in time.
Z. Li, D. Kovalev, X. Qian, and P. Richtárik, Acceleration for compressed gradient descent in distributed and federated optimization , arXiv preprint arXiv:2002.11364 (2020)
Original
2020
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

C. Ma, J. Konečný, M. Jaggi, V. Smith, M.I. Jordan, P. Richtárik, and M. Takáč, Distributed optimization with arbitrary local solvers , Optimization Methods and Software 32 (2017), pp. 813–848

2017

Cited alongside, same era.

W. Wen, C. Xu, F. Yan, C. Wu, Y. Wang, Y. Chen, and H. Li, Terngrad: Ternary gradients to reduce communication in distributed deep learning , in Advances in Neural Information Processing Systems . 2017, pp. 1509–1519

2017

Cited alongside, same era.

J. Bernstein, Y.X. Wang, K. Azizzadenesheli, and A. Anandkumar, signSGD: Compressed Optimisation for Non-Convex Problems , in Proceedings of the 35th International Conference on Machine Learning , J. Dy and A. Krause, eds., Proceedings of Machine Learning Research Vol. 80, 10–15 Jul, Stockholmsmässan, Stockholm Sweden. PMLR, 2018, pp. 560–569

2018

Cited alongside, same era.

M. Jahani, X. He, C. Ma, A. Mokhtari, D. Mudigere, A. Ribeiro, and M. Takáč, Efficient distributed hessian free algorithm for large-scale empirical risk minimization via accumulating sample strategy , arXiv:1810.11507 (2018)

Original

2018

Cited alongside, same era.

S. Khirirat, H.R. Feyzmahdavian, and M. Johansson, Distributed learning with compressed gradients , arXiv preprint arXiv:1806.06573 (2018)

Original

2018

Cited alongside, same era.

V. Smith, S. Forte, C. Ma, M. Takáč, M.I. Jordan, and M. Jaggi, CoCoA: A general framework for communication-efficient distributed optimization , Journal of Machine Learning Research 18 (2018), pp. 1–49

2018

Cited alongside, same era.

C. Philippenko and A. Dieuleveut, Bidirectional compression in heterogeneous settings for distributed or federated learning with partial participation: tight convergence guarantees , arXiv preprint arXiv:2006.14591 (2020)

Original

2020

Closest in time.

D. Kovalev, A. Koloskova, M. Jaggi, P. Richtarik, and S. Stich, A linearly convergent algorithm for decentralized optimization: Sending less bits for free! , in International Conference on Artificial Intelligence and Statistics . PMLR, 2021, pp. 4087–4095

2021

Closest in time.

C. Philippenko and A. Dieuleveut, Preserved central model for faster bidirectional compression in distributed settings , Advances in Neural Information Processing Systems 34 (2021), pp. 2387–2399

2021

Closest in time.

M. Safaryan, R. Islamov, X. Qian, and P. Richtárik, Fednl: Making newton-type methods applicable to federated learning , arXiv preprint arXiv:2106.02969 (2021)

Original

2021

Closest in time.

H. Zhu and Q. Ling, Broadcast: Reducing both stochastic and compression noise to robustify communication-efficient federated learning , arXiv preprint arXiv:2104.06685 (2021)

Original

2021

Closest in time.

A. Beznosikov, E. Gorbunov, H. Berard, and N. Loizou, Stochastic gradient descent-ascent: Unified theory and new efficient methods , arXiv preprint arXiv:2202.07262 (2022)

Original

2022

Closest in time.