2020

Communication-Efficient Distributed Deep Learning: A Comprehensive Survey

Tang, Zhenheng, Shi, Shaohuai, Wang, Wei et al.

Understand

Distributed deep learning (DL) has become prevalent in recent years to reduce training time by leveraging multiple computing devices (e.g., GPUs/TPUs) due to larger models and datasets.

However, system scalability is limited by communication becoming the performance bottleneck.
Addressing this communication issue has become a prominent research topic.
In this paper, we provide a comprehensive survey of the communication-efficient distributed training algorithms, focusing on both system-level and algorithmic-level optimizations.

Built on

Television by pulse code modulation
W. M. Goodall · 1951
Earlier work this paper cites.
Picture coding using pseudo-random noise
L. Roberts · 1962
Earlier work this paper cites.
A bridging model for parallel computation
L. G. Valiant · 1990
Earlier work this paper cites.
Probabilistic rounding in neural network learning with limited precision
M. Höhfeld and S. E. Fahlman · 1992
Earlier work this paper cites.
Learning long-term dependencies with gradient descent is difficult
Y. Bengio, P. Simard, and P. Frasconi · 1994
Earlier work this paper cites.
Bulk synchronous parallel computing-a paradigm for transportable software
T. Cheatham, A. Fahmy, D. C. Stefanescu, and L. G. Valiant · 1995
Earlier work this paper cites.
Bulk synchronous parallel: practical experience with a model for parallel computing
R. Krizanc and A. Saarimaki · 1996
Earlier work this paper cites.
Efficient algorithms for all-to-all communications in multiport message-passing systems
J. Bruck, Ching-Tien Ho, S. Kipnis, E. Upfal, and D. Weathersby · 1997
Earlier work this paper cites.
A tonotopic artificial neural network architecture for phoneme probability estimation
N. Strom · 1997
Earlier work this paper cites.
Sparse connection and pruning in large dynamic artificial neural networks
N. Ström · 1997
Earlier work this paper cites.
On the momentum term in gradient descent learning algorithms
N. Qian · 1999
Earlier work this paper cites.
Focusing Solutions for Data Mining: Analytical Studies and Experimental Results in Real-world Domains
T. Reinartz · 1999
Earlier work this paper cites.
Connection-level analysis and modeling of network traffic
S. Sarvotham, R. Riedi, and R. Baraniuk · 2001
Earlier work this paper cites.
Finding frequent items in data streams
M. Charikar, K. Chen, and M. Farach-Colton · 2002
Earlier work this paper cites.
Gossip-based computation of aggregate information
D. Kempe, A. Dobra, and J. Gehrke · 2003
Earlier work this paper cites.
Fast linear iterations for distributed averaging
Lin Xiao and S. Boyd · 2003
Earlier work this paper cites.
Optimization of collective reduction operations
R. Rabenseifner · 2004
Earlier work this paper cites.
Gossip algorithms: design, analysis and applications
S. Boyd, A. Ghosh, B. Prabhakar, and D. Shah · 2005
Earlier work this paper cites.
Optimization of collective communication operations in mpich
R. Thakur, R. Rabenseifner, and W. Gropp · 2005
Earlier work this paper cites.
Randomized gossip algorithms
S. Boyd, A. Ghosh, B. Prabhakar, and D. Shah · 2006
Earlier work this paper cites.
Consensus and cooperation in networked multi-agent systems
R. Olfati-Saber, J. A. Fax, and R. M. Murray · 2007
Earlier work this paper cites.
A distributed consensus protocol for clock synchronization in wireless sensor network
L. Schenato and G. Gamba · 2007
Earlier work this paper cites.
Mpi for python: Performance improvements and mpi-2 extensions
L. Dalcín, R. Paz, M. Storti, and J. D’Elía · 2008
Earlier work this paper cites.
Randomized consensus algorithms over large scale networks
F. Fagnani and S. Zampieri · 2008
Earlier work this paper cites.
Distributed stochastic subgradient projection algorithms for convex optimization
S. S. Ram, A. Nedic, and V. V. Veeravalli · 2008
Earlier work this paper cites.
Imagenet: A large-scale hierarchical image database
J. Deng, W. Dong, R. Socher, L. Li, Kai Li, and Li Fei-Fei · 2009
Earlier work this paper cites.
BCube: a high performance, server-centric network architecture for modular data centers
C. Guo, G. Lu, D. Li, H. Wu, X. Zhang, Y. Shi, C. Tian, Y. Zhang, and S. Lu · 2009
Earlier work this paper cites.
Sparse online learning via truncated gradient
J. Langford, L. Li, and T. Zhang · 2009
Earlier work this paper cites.
Efficient large-scale distributed training of conditional maximum entropy models
R. Mcdonald, M. Mohri, N. Silberman, D. Walker, and G. S. Mann · 2009
Earlier work this paper cites.
Distributed subgradient methods for multi-agent optimization
A. Nedic and A. Ozdaglar · 2009
Earlier work this paper cites.
Two-tree algorithms for full bandwidth broadcast, reduction and scan
P. Sanders, J. Speck, and J. L. Träff · 2009
Earlier work this paper cites.
Gossip consensus algorithms via quantized communication
R. Carli, F. Fagnani, P. Frasca, and S. Zampieri · 2010
Earlier work this paper cites.
Toward performance models of mpi implementations for understanding application scaling issues
T. Hoefler, W. Gropp, R. Thakur, and J. L. Träff · 2010
Earlier work this paper cites.
Cifar-10 (canadian institute for advanced research)
A. Krizhevsky, V. Nair, and G. Hinton · 2010
Earlier work this paper cites.
Distributed training strategies for the structured perceptron
R. T. McDonald, K. B. Hall, and G. Mann · 2010
Earlier work this paper cites.
Piccolo: Building fast, distributed programs with partitioned tables
R. Power and J. Li · 2010
Earlier work this paper cites.
An architecture for parallel topic models
A. Smola and S. Narayanamurthy · 2010
Earlier work this paper cites.
Parallelized stochastic gradient descent
M. A. Zinkevich, M. Weimer, A. Smola, and L. Li · 2010
Earlier work this paper cites.
Parallel coordinate descent for l1-regularized loss minimization
J. K. Bradley, A. Kyrola, D. Bickson, and C. Guestrin · 2011
Earlier work this paper cites.
Hogwild!: A lock-free approach to parallelizing stochastic gradient descent
F. Niu, B. Recht, C. Re, and S. J. Wright · 2011
Earlier work this paper cites.
Improving the speed of neural networks on cpus
V. Vanhoucke, A. Senior, and M. Z. Mao · 2011
Earlier work this paper cites.
Scalable inference in latent variable models
A. Ahmed, M. Aly, J. Gonzalez, S. Narayanamurthy, and A. J. Smola · 2012
Earlier work this paper cites.
Large scale distributed deep networks
J. Dean, G. Corrado, R. Monga, K. Chen, M. Devin, M. Mao, M. Ranzato, A. Senior, P. Tucker, K. Yang, et al · 2012
Earlier work this paper cites.
Distributed graphlab: A framework for machine learning and data mining in the cloud
Y. Low, D. Bickson, J. Gonzalez, C. Guestrin, A. Kyrola, and J. M. Hellerstein · 2012
Earlier work this paper cites.
More effective distributed ml via a stale synchronous parallel parameter server
Q. Ho, J. Cipar, H. Cui, J. K. Kim, S. Lee, P. B. Gibbons, G. A. Gibson, G. R. Ganger, and E. P. Xing · 2013
Earlier work this paper cites.
Asynchronous distributed optimization using a randomized alternating direction method of multipliers
F. Iutzeler, P. Bianchi, P. Ciblat, and W. Hachem · 2013
Earlier work this paper cites.
Parameter server for distributed machine learning
M. Li, L. Zhou, Z. Yang, A. Q. Li, F. Xia, D. G. Andersen, and A. J. Smola · 2013
Earlier work this paper cites.
Distributed delayed proximal gradient methods
D. G. A. M. Li and A. Smola · 2013
Earlier work this paper cites.
D-admm: A communication-efficient distributed algorithm for separable optimization
J. F. C. Mota, J. M. F. Xavier, P. M. Q. Aguiar, and M. Püschel · 2013
Earlier work this paper cites.
A survey of methods for distributed machine learning
D. Peteiro-Barral and B. Guijarro-Berdiñas · 2013
Earlier work this paper cites.
On the o(1=k) convergence of asynchronous distributed alternating direction method of multipliers
E. Wei and A. E. Ozdaglar · 2013
Earlier work this paper cites.
A block coordinate descent method for regularized multiconvex optimization with applications to nonnegative tensor factorization and completion
Y. Xu and W. Yin · 2013
Earlier work this paper cites.
Distributed stochastic gradient mcmc
S. Ahn, B. Shahbaba, and M. Welling · 2014
Earlier work this paper cites.
cuDNN: Efficient primitives for deep learning
Original
S. Chetlur, C. Woolley, P. Vandermersch, J. Cohen, J. Tran, B. Catanzaro, and E. Shelhamer · 2014
Earlier work this paper cites.
Project adam: Building an efficient and scalable deep learning training system
T. Chilimbi, Y. Suzue, J. Apacible, and K. Kalyanaraman · 2014
Earlier work this paper cites.
Exploiting bounded staleness to speed up big data analytics
H. Cui, J. Cipar, Q. Ho, J. K. Kim, S. Lee, A. Kumar, J. Wei, W. Dai, G. R. Ganger, P. B. Gibbons, G. A. Gibson, and E. P. Xing · 2014
Earlier work this paper cites.
Exploiting iterative-ness for parallel ml computations
H. Cui, A. Tumanov, J. Wei, L. Xu, W. Dai, J. Haber-Kucharsky, Q. Ho, G. R. Ganger, P. B. Gibbons, G. A. Gibson, and E. P. Xing · 2014
Earlier work this paper cites.
One weird trick for parallelizing convolutional neural networks
Original
A. Krizhevsky · 2014
Earlier work this paper cites.
Scaling distributed machine learning with the parameter server
M. Li, D. G. Andersen, J. W. Park, A. J. Smola, A. Ahmed, V. Josifovski, J. Long, E. J. Shekita, and B.-Y. Su · 2014
Earlier work this paper cites.
Communication efficient distributed machine learning with the parameter server
M. Li, D. G. Andersen, A. Smola, and K. Yu · 2014
Earlier work this paper cites.
1-bit stochastic gradient descent and its application to data-parallel distributed training of speech DNNs
F. Seide, H. Fu, J. Droppo, G. Li, and D. Yu · 2014
Earlier work this paper cites.
Dimmwitted: A study of main-memory statistical analytics
C. Zhang and C. Ré · 2014
Earlier work this paper cites.
Asynchronous distributed admm for consensus optimization
R. Zhang and J. T. Kwok · 2014
Earlier work this paper cites.
Improving deep neural network acoustic models using generalized maxout networks
X. Zhang, J. Trmal, D. Povey, and S. Khudanpur · 2014
Earlier work this paper cites.
Mariana: Tencent deep learning platform and its applications
Y. Zou, X. Jin, Y. Li, Z. Guo, E. Wang, and B. Xiao · 2014
Earlier work this paper cites.
Deep learning with limited numerical precision
S. Gupta, A. Agrawal, K. Gopalakrishnan, and P. Narayanan · 2015
Earlier work this paper cites.
Passcode: Parallel asynchronous stochastic dual co-ordinate descent
C.-J. Hsieh, H.-F. Yu, and I. S. Dhillon · 2015
Earlier work this paper cites.
Asynchronous parallel stochastic gradient for nonconvex optimization
X. Lian, Y. Huang, Y. Li, and J. Liu · 2015
Earlier work this paper cites.
Optimizing neural networks with kronecker-factored approximate curvature
J. Martens and R. Grosse · 2015
Earlier work this paper cites.
Singa: A distributed deep learning platform
B. C. Ooi, K.-L. Tan, S. Wang, W. Wang, Q. Cai, G. Chen, J. Gao, Z. Luo, A. K. Tung, Y. Wang, Z. Xie, M. Zhang, and K. Zheng · 2015
Earlier work this paper cites.
Quartz: Randomized dual coordinate ascent with arbitrary sampling
Z. Qu, P. Richtárik, and T. Zhang · 2015
Earlier work this paper cites.
Multi-agent mirror descent for decentralized stochastic optimization
M. Rabbat · 2015
Earlier work this paper cites.
Imagenet large scale visual recognition challenge
O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Z. Huang, A. Karpathy, A. Khosla, M. Bernstein, A. C. Berg, and L. Fei-Fei · 2015
Earlier work this paper cites.
Taming the wild: A unified analysis of hogwild-style algorithms
C. D. Sa, C. Zhang, K. Olukotun, and C. Ré · 2015
Earlier work this paper cites.
Scalable distributed DNN training using commodity GPU cloud computing
N. Strom · 2015
Earlier work this paper cites.
Petuum: A new platform for distributed machine learning on big data
E. P. Xing, Q. Ho, W. Dai, J. K. Kim, J. Wei, S. Lee, X. Zheng, P. Xie, A. Kumar, and Y. Yu · 2015
Earlier work this paper cites.
Deep learning with elastic averaging sgd
S. Zhang, A. Choromanska, and Y. LeCun · 2015
Earlier work this paper cites.
Tensorflow: A system for large-scale machine learning
M. Abadi, P. Barham, J. Chen, Z. Chen, A. Davis, J. Dean, M. Devin, S. Ghemawat, G. Irving, M. Isard, M. Kudlur, et al · 2016
Earlier work this paper cites.
Qsgd: Randomized quantization for communication-optimal stochastic gradient descent
Original
D. Alistarh, J. Li, R. Tomioka, and M. Vojnovic · 2016
Earlier work this paper cites.
Deep speech 2: End-to-end speech recognition in english and mandarin
D. Amodei, S. Ananthanarayanan, R. Anubhai, et al · 2016
Earlier work this paper cites.
Layer normalization
Original
J. Ba, J. R. Kiros, and G. E. Hinton · 2016
Earlier work this paper cites.
On data dependence in distributed stochastic optimization
A. S. Bijral, A. D. Sarwate, and N. Srebro · 2016
Earlier work this paper cites.
Optimization methods for large-scale machine learning
L. Bottou, F. E. Curtis, and J. Nocedal · 2016
Earlier work this paper cites.
Revisiting distributed synchronous sgd
J. Chen, R. Monga, S. Bengio, and R. Jozefowicz · 2016
Earlier work this paper cites.
Scalable training of deep learning machines by incremental block training with intra-block parallel optimization and blockwise model-update filtering
K. Chen and Q. Huo · 2016
Earlier work this paper cites.
Gossip dual averaging for decentralized optimization of pairwise functions
I. Colin, A. Bellet, J. Salmon, and S. Clémençon · 2016
Earlier work this paper cites.
Geeps: Scalable deep learning on distributed gpus with a gpu-specialized parameter server
H. Cui, H. Zhang, G. R. Ganger, P. B. Gibbons, and E. P. Xing · 2016
Earlier work this paper cites.
Communication quantization for data-parallel training of deep neural networks
N. Dryden, S. A. Jacobs, T. Moon, and B. Van Essen · 2016
Earlier work this paper cites.

Similar

Deep Learning
I. Goodfellow, Y. Bengio, and A. Courville · 2016
Cited alongside, same era.
Deep residual learning for image recognition
K. He, X. Zhang, S. Ren, and J. Sun · 2016
Cited alongside, same era.
Federated learning: Strategies for improving communication efficiency
J. Konečný, H. B. McMahan, F. X. Yu, P. Richtarik, A. T. Suresh, and D. Bacon · 2016
Cited alongside, same era.
Distributed semi-stochastic optimization with quantization refinement
N. McGlohon and S. Patterson · 2016
Cited alongside, same era.
Asynchronous accelerated stochastic gradient descent
Q. Meng, W. Chen, J. Yu, T. Wang, Z.-M. Ma, and T.-Y. Liu · 2016
Cited alongside, same era.
Sparknet: Training deep networks in spark
P. Moritz, R. Nishihara, I. Stoica, and M. I. Jordan · 2016
Cited alongside, same era.
Stochastic gradient-push for strongly convex functions on time-varying directed graphs
A. Nedić and A. Olshevsky · 2016
Cited alongside, same era.
How to scale distributed deep learning?
F. N. I. P. H. Jin, Q. Yuan and K. Keutzer · 2016
Cited alongside, same era.
Benchmarking state-of-the-art deep learning software tools
S. Shi, Q. Wang, P. Xu, and X. Chu · 2016
Cited alongside, same era.
Topic modeling and visualization for big data in social sciences
N. Sukhija, M. Tatineni, N. Brown, M. V. Moer, P. Rodriguez, and S. Callicott · 2016
Cited alongside, same era.
Strategies and principles of distributed machine learning on big data
E. P. Xing, Q. Ho, P. Xie, and D. Wei · 2016
Cited alongside, same era.
Parallel sgd: When does averaging help?
Original
J. Zhang, C. D. Sa, I. Mitliagkas, and C. Ré · 2016
Cited alongside, same era.

Then

Stochastic gradient push for distributed deep learning
M. Assran, N. Loizou, N. Ballas, and M. Rabbat · 2019
Later among the works it cites.
Qsparse-local-SGD: Distributed SGD with quantization, sparsification and local computations
D. Basu, D. Data, C. Karakus, and S. Diggavi · 2019
Later among the works it cites.
Demystifying parallel and distributed deep learning: An in-depth concurrency analysis
T. Ben-Nun and T. Hoefler · 2019
Later among the works it cites.
Round-robin synchronization: Mitigating communication bottlenecks in parameter servers
C. Chen, W. Wang, and B. Li · 2019
Later among the works it cites.
On the ineffectiveness of variance reduced optimization for deep learning
A. Defazio and L. Bottou · 2019
Later among the works it cites.
BERT: Pre-training of deep bidirectional transformers for language understanding
J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova · 2019
Later among the works it cites.
Communication trade-offs for local-sgd with large step size
A. Dieuleveut and K. K. Patel · 2019
Later among the works it cites.
Local sgd with periodic averaging: Tighter analysis and adaptive synchronization
F. Haddadpour, M. M. Kamani, M. Mahdavi, and V. Cadambe · 2019
Later among the works it cites.
Natural compression for distributed deep learning
Original
S. Horvath, C.-Y. Ho, L. Horvath, A. N. Sahu, M. Canini, and P. Richtárik · 2019
Later among the works it cites.
Gpipe: Efficient training of giant neural networks using pipeline parallelism
Y. Huang, Y. Cheng, A. Bapna, O. Firat, D. Chen, M. Chen, H. Lee, J. Ngiam, Q. V. Le, Y. Wu, et al · 2019
Later among the works it cites.
Communication-efficient distributed sgd with sketching
N. Ivkin, D. Rothchild, E. Ullah, V. braverman, I. Stoica, and R. Arora · 2019
Later among the works it cites.
Advances and open problems in federated learning
Original
P. Kairouz, H. B. McMahan, B. Avent, A. Bellet, M. Bennis, A. N. Bhagoji, K. Bonawitz, Z. Charles, G. Cormode, R. Cummings, et al · 2019
Later among the works it cites.
Error feedback fixes SignSGD and other gradient compression schemes
S. P. Karimireddy, Q. Rebjock, S. Stich, and M. Jaggi · 2019
Later among the works it cites.
Decentralized deep learning with arbitrary communication compression
Original
A. Koloskova, T. Lin, S. U. Stich, and M. Jaggi · 2019
Later among the works it cites.
Decentralized stochastic optimization and gossip algorithms with compressed communication
A. Koloskova, S. Stich, and M. Jaggi · 2019
Later among the works it cites.
Automating system configuration of distributed machine learning
W. Lee, Y. Lee, J. S. Jeong, G. Yu, J. Y. Kim, H. J. Park, B. Jeon, W. Song, G. Kim, M. Weimer, B. Cho, and B. Chun · 2019
Later among the works it cites.
3lc: Lightweight and effective traffic compression for distributed machine learning
Original
H. Lim, D. G. Andersen, and M. Kaminsky · 2019
Later among the works it cites.
Distributed learning with compressed gradient differences
Original
K. Mishchenko, E. A. Gorbunov, M. Takác, and P. Richtárik · 2019
Later among the works it cites.
99% of parallel optimization is inevitably a waste of time
Original
K. Mishchenko, F. Hanzely, and P. Richtárik · 2019
Later among the works it cites.
A generic communication scheduler for distributed DNN training acceleration
Y. Peng, Y. Zhu, Y. Chen, Y. Bao, B. Yi, C. Lan, C. Wu, and C. Guo · 2019
Later among the works it cites.
Robust and communication-efficient collaborative learning
A. Reisizadeh, H. Taheri, A. Mokhtari, H. Hassani, and R. Pedarsani · 2019
Later among the works it cites.
Robust and communication-efficient federated learning from non-iid data
Original
F. Sattler, S. Wiedemann, K. Müller, and W. Samek · 2019
Later among the works it cites.
Understanding top-k sparsification in distributed deep learning
Original
S. Shi, X. Chu, K. C. Cheung, and S. See · 2019
Later among the works it cites.
MG-WFBP: Efficient data communication for distributed synchronous SGD algorithms
S. Shi, X. Chu, and B. Li · 2019
Later among the works it cites.
A distributed synchronous SGD algorithm with global top-k sparsification for low bandwidth networks
S. Shi, Q. Wang, K. Zhao, Z. Tang, Y. Wang, X. Huang, and X. Chu · 2019
Later among the works it cites.
A convergence analysis of distributed SGD with communication-efficient gradient sparsification
S. Shi, K. Zhao, Q. Wang, Z. Tang, and X. Chu · 2019
Later among the works it cites.
Megatron-lm: Training multi-billion parameter language models using model parallelism
Original
M. Shoeybi, M. Patwary, R. Puri, P. LeGresley, J. Casper, and B. Catanzaro · 2019
Later among the works it cites.
Local SGD converges fast and communicates little
S. U. Stich · 2019
Later among the works it cites.
Communication-efficient distributed learning via lazily aggregated quantized gradients
J. Sun, T. Chen, G. Giannakis, and Z. Yang · 2019
Later among the works it cites.
Exhaustive study of hierarchical allreduce patterns for large messages between gpus
Y. Ueno and R. Yokota · 2019
Later among the works it cites.
CMFL: mitigating communication overhead for federated learning
L. Wang, W. Wang, and B. Li · 2019
Later among the works it cites.
Impact of network topology on the performance of dml: Theoretical analysis and practical factors
S. Wang, D. Li, J. Geng, Y. Gu, and Y. Cheng · 2019
Later among the works it cites.
Benchmarking the performance and power of AI accelerators for AI training
Original
Y. Wang, Q. Wang, S. Shi, X. He, Z. Tang, K. Zhao, and X. Chu · 2019
Later among the works it cites.
On the computation and communication complexity of parallel SGD with dynamic batch sizes for stochastic non-convex optimization
H. Yu and R. Jin · 2019
Later among the works it cites.
On the linear speedup analysis of communication efficient momentum SGD for distributed non-convex optimization
H. Yu, R. Jin, and S. Yang · 2019
Later among the works it cites.
Double quantization for communication-efficient distributed optimization
Y. Yu, J. Wu, and L. Huang · 2019
Later among the works it cites.
Global convergence of block coordinate descent in deep learning
J. Zeng, T. T.-K. Lau, S. Lin, and Y. Yao · 2019
Later among the works it cites.
Global momentum compression for sparse communication in distributed sgd
Original
S.-Y. Zhao, Y. Xie, H. Gao, and W.-J. Li · 2019
Later among the works it cites.
Language models are few-shot learners
Original
T. B. Brown, B. Mann, N. Ryder, M. Subbiah, J. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, et al · 2020
Closest in time.
On the discrepancy between the theoretical analysis and practical implementations of compressed communication for distributed deep learning
A. Dutta, E. H. Bergou, A. M. Abdelmoniem, C.-Y. Ho, A. N. Sahu, M. Canini, and P. Kalnis · 2020
Closest in time.
Adaptive gradient quantization for data-parallel sgd
F. Faghri, I. Tabrizian, I. Markov, D. Alistarh, D. M. Roy, and A. Ramezani-Kebrya · 2020
Closest in time.
Adaptive gradient sparsification for efficient federated learning: An online learning approach
Original
P. Han, S. Wang, and K. K. Leung · 2020
Closest in time.
Fedml: A research library and benchmark for federated machine learning
Original
C. He, S. Li, J. So, M. Zhang, H. Wang, X. Wang, P. Vepakomma, A. Singh, H. Qiu, L. Shen, P. Zhao, Y. Kang, Y. Liu, R. Raskar, Q. Yang, M. Annavaram, and S. Avestimehr · 2020
Closest in time.
Don’t use large mini-batches, use local sgd
T. Lin, S. U. Stich, K. K. Patel, and M. Jaggi · 2020
Closest in time.
PLink: Efficient cloud-based training with topology-aware dynamic hierarchical aggregation
L. Luo, P. West, J. Nelson, A. Krishnamurthy, and L. Ceze · 2020
Closest in time.
A quantitative survey of communication optimizations in distributed deep learning
S. Shi, Z. Tang, X. Chu, C. Liu, W. Wang, and B. Li · 2020
Closest in time.
Layer-wise adaptive gradient sparsification for distributed deep learning with convergence guarantees
S. Shi, Z. Tang, Q. Wang, K. Zhao, and X. Chu · 2020
Closest in time.
Communication-efficient distributed deep learning with merged gradient sparsification on gpus
S. Shi, Q. Wang, X. Chu, B. Li, Y. Qin, R. Liu, and X. Zhao · 2020
Closest in time.
Communication-efficient decentralized learning with sparsification and adaptive peer selection
Original
Z. Tang, S. Shi, and X. Chu · 2020
Closest in time.
Blink: Fast and generic collectives for distributed ML
G. Wang, S. Venkataraman, A. Phanishayee, J. Thelin, N. Devanur, and I. Stoica · 2020
Closest in time.
Communication contention aware scheduling of multiple deep learning training jobs
Original
Q. Wang, S. Shi, C. Wang, and X. Chu · 2020
Closest in time.
Optimizing batched winograd convolution on GPUs
D. Yan, W. Wang, and X. Chu · 2020
Closest in time.
A survey of deep learning techniques for neural machine translation
Original
S. Yang, Y. Wang, and X. Chu · 2020
Closest in time.
Large batch optimization for deep learning: Training BERT in 76 minutes
Y. You, J. Li, S. Reddi, J. Hseu, S. Kumar, S. Bhojanapalli, X. Song, J. Demmel, K. Keutzer, and C.-J. Hsieh · 2020
Closest in time.
Distributed hierarchical gpu parameter server for massive scale deep learning ads systems
W. Zhao, D. Xie, R. Jia, Y. Qian, R. Ding, M. Sun, and P. Li · 2020
Closest in time.
Efficient sparse collective communication and its application to accelerate distributed deep learning
J. Fei, C.-Y. Ho, A. N. Sahu, M. Canini, and A. Sapio · 2021
Closest in time.
Adaptive quantization of model updates for communication-efficient federated learning
D. Jhunjhunwala, A. Gadhikar, G. Joshi, and Y. C. Eldar · 2021
Closest in time.
{GS}hard: Scaling giant models with conditional computation and automatic sharding
D. Lepikhin, H. Lee, Y. Xu, D. Chen, O. Firat, Y. Huang, M. Krikun, N. Shazeer, and Z. Chen · 2021
Closest in time.
Chimera: efficiently training large-scale neural networks with bidirectional pipelines
S. Li and T. Hoefler · 2021
Closest in time.
An efficient statistical-based gradient compression technique for distributed training systems
A. M Abdelmoniem, A. Elzanaty, M.-S. Alouini, and M. Canini · 2021
Closest in time.
Ef21: A new, simpler, theoretically better, and practically faster error feedback
P. Richtárik, I. Sokolov, and I. Fatkhullin · 2021
Closest in time.
Scaling distributed machine learning with In-Network aggregation
A. Sapio, M. Canini, C.-Y. Ho, J. Nelson, P. Kalnis, C. Kim, A. Krishnamurthy, M. Moshref, D. Ports, and P. Richtarik · 2021
Closest in time.
MG-WFBP: Merging gradients wisely for efficient communication in distributed deep learning
S. Shi, X. Chu, and B. Li · 2021
Closest in time.
Accelerating distributed k-fac with smart parallelism of computing and communication tasks
S. Shi, L. Zhang, and B. Li · 2021
Closest in time.
Towards scalable distributed training of deep learning on public cloud clusters
S. Shi, X. Zhou, S. Song, X. Wang, Z. Zhu, X. Huang, X. Jiang, F. Zhou, Z. Guo, L. Xie, et al · 2021
Closest in time.
Lena: Communication-efficient distributed learning with self-triggered gradient uploads
H. Shokri Ghadikolaei, S. Stich, and M. Jaggi · 2021
Closest in time.
Communication-efficient SGD: From local SGD to one-shot averaging
A. Spiridonoff, A. Olshevsky, and I. Paschalidis · 2021
Closest in time.
Elasticl: Elastic quantization for communication efficient collaborative learning in iot
B. Sudharsan, D. Sheth, S. Arya, F. Rollo, P. Yadav, P. Patel, J. G. Breslin, and M. I. Ali · 2021
Closest in time.
1-bit adam: Communication efficient large-scale training with adam’s convergence speed
H. Tang, S. Gan, A. A. Awan, S. Rajbhandari, C. Li, X. Lian, J. Liu, C. Zhang, and Y. He · 2021
Closest in time.
On the utility of gradient compression in distributed training systems
S. Agarwal, H. Wang, S. Venkataraman, and D. Papailiopoulos · 2022
Closest in time.
Pisces: efficient federated learning via guided asynchronous training
Z. Jiang, W. Wang, B. Li, and B. Li · 2022
Closest in time.
Near-optimal sparse allreduce for distributed deep learning
S. Li and T. Hoefler · 2022
Closest in time.
On distributed adaptive optimization with gradient compression
X. Li, B. Karimi, and P. Li · 2022
Closest in time.
Asynchronous sgd beats minibatch sgd under arbitrary delays
K. Mishchenko, F. Bach, M. Even, and B. E. Woodworth · 2022
Closest in time.
3PC: Three point compressors for communication-efficient distributed training and a better theory for lazy aggregation
P. Richtarik, I. Sokolov, E. Gasanov, I. Fatkhullin, Z. Li, and E. Gorbunov · 2022
Closest in time.
Accelerating collective communication in data parallel training across deep learning frameworks
J. Romero, J. Yin, N. Laanait, B. Xie, M. T. Young, S. Treichler, V. Starchenko, A. Borisevich, A. Sergeev, and M. Matheson · 2022
Closest in time.
Gossipfl: A decentralized federated learning framework with sparsified and adaptive communication
Z. Tang, S. Shi, B. Li, and X. Chu · 2022
Closest in time.
Virtual homogeneity learning: Defending against data heterogeneity in federated learning
Z. Tang, Y. Zhang, S. Shi, X. He, B. Han, and X. Chu · 2022
Closest in time.
Error-compensated sparsification for communication-efficient decentralized training in edge environment
H. Wang, S. Guo, Z. Qu, R. Li, and Z. Liu · 2022
Closest in time.
Communication-compressed adaptive gradient method for distributed nonconvex optimization
Y. Wang, L. Lin, and J. Chen · 2022
Closest in time.
Detached error feedback for distributed SGD with random sparsification
A. Xu and H. Huang · 2022
Closest in time.
Ac-sgd: Adaptively compressed sgd for communication-efficient distributed learning
G. Yan, T. Li, S.-L. Huang, T. Lan, and L. Song · 2022
Closest in time.
Mipd: An adaptive gradient sparsification framework for distributed dnns training
Z. Zhang and C. Wang · 2022
Closest in time.
PipeMoE: Accelerating mixture-of-experts through adaptive pipelining
S. Shi, X. Pan, X. Chu, and B. Li · 2023
Closest in time.
Fedml parrot: A scalable federated learning system via heterogeneity-aware scheduling on sequential and hierarchical training
Original
Z. Tang, X. Chu, R. Y. Ran, S. Lee, S. Shi, Y. Zhang, Y. Wang, A. Q. Liang, S. Avestimehr, and C. He · 2023
Closest in time.
Accelerating distributed deep learning with fine-grained all-reduce pipelining
L. Zhang, S. Shi, X. Chu, W. Wang, B. Li, and C. Liu · 2023
Closest in time.
Eva: Practical second-order optimization with kronecker-vectorized approximation
L. Zhang, S. Shi, and B. Li · 2023
Closest in time.
Evaluation and optimization of gradient compression for distributed deep learning
L. Zhang, L. Zhang, S. Shi, X. Chu, and B. Li · 2023
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…