CGX: Adaptive System Support for Communication-Efficient Deep Learning

Transformer-xl: Attentive language models beyond a fixed-length context

Original

Zihang Dai, Zhilin Yang, Yiming Yang, Jaime Carbonell, Quoc V Le, and Ruslan Salakhutdinov. 2019 · 1901

Earlier work this paper cites.

Large batch optimization for deep learning: Training bert in 76 minutes

Original

Yang You, Jing Li, Sashank Reddi, Jonathan Hseu, Sanjiv Kumar, Srinadh Bhojanapalli, Xiaodan Song, James Demmel, Kurt Keutzer, and Cho-Jui Hsieh. 2019 · 1904

Earlier work this paper cites.

Priority-based parameter propagation for distributed DNN training

Original

Anand Jayarajan, Jinliang Wei, Garth Gibson, Alexandra Fedorova, and Gennady Pekhimenko. 2019 · 1905

Earlier work this paper cites.

Some Methods for Classification and Analysis of MultiVariate Observations. In Proc. of the fifth Berkeley Symposium on Mathematical Statistics and Probability (Berkeley, CA, USA, June 21-July 18 1965), Vol. 1. University of California Press, 281–297

J. B. MacQueen. 1967 · 1965

Earlier work this paper cites.

Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition (Miami, FL, June 20 - 25, 2009). IEEE, 248–255

Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. 2009 · 2009

Earlier work this paper cites.

Project Adam: Building an Efficient and Scalable Deep Learning Training System. In Proceedings of the 11th USENIX Conference on Operating Systems Design and Implementation (Broomfield, CO, USA, October 6-8, 2014), Vol. 14. 571–582

Trishul M Chilimbi, Yutaka Suzue, Johnson Apacible, and Karthik Kalyanaraman. 2014 · 2014

Earlier work this paper cites.

Scaling distributed machine learning with the parameter server. In Proceedings 11th { \{ USENIX } \} Symposium on Operating Systems Design and Implementation ( { \{ OSDI } \} 14) (Broomfield, CO, USA, October 6-8, 2014). 583–598

Mu Li, David G Andersen, Jun Woo Park, Alexander J Smola, Amr Ahmed, Vanja Josifovski, James Long, Eugene J Shekita, and Bor-Yiing Su. 2014 · 2014

Earlier work this paper cites.

1-bit stochastic gradient descent and its application to data-parallel distributed training of speech dnns. In Fifteenth Annual Conference of the International Speech Communication Association (Singapore, September 14-18, 2014)

Frank Seide, Hao Fu, Jasha Droppo, Gang Li, and Dong Yu. 2014 · 2014

Earlier work this paper cites.

Very deep convolutional networks for large-scale image recognition

Original

Karen Simonyan and Andrew Zisserman. 2014 · 2014

Earlier work this paper cites.

Scalable distributed DNN training using commodity GPU cloud computing. In Sixteenth Annual Conference of the International Speech Communication Association (Dresden, Germany, September 6-10, 2015)

Nikko Strom. 2015 · 2015

Earlier work this paper cites.

Tensorflow: A system for large-scale machine learning. In Proceedings of the 12th USENIX Conference on Operating Systems Design and Implementation (Savannah, GA, USA, November 2 - 4, 2016) (OSDI’16) . USENIX Association, USA, 265–283

Martín Abadi, Paul Barham, Jianmin Chen, Zhifeng Chen, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Geoffrey Irving, Michael Isard, et al · 2016

Earlier work this paper cites.

Communication quantization for data-parallel training of deep neural networks. In Proceedings of the Workshop on Machine Learning in High Performance Computing Environments (Salt Lake City, UT, USA, November 14 2016). IEEE Press, 1–8

Nikoli Dryden, Sam Ade Jacobs, Tim Moon, and Brian Van Essen. 2016 · 2016

Earlier work this paper cites.

QSGD: Communication-efficient SGD via gradient quantization and encoding. In Advances in Neural Information Processing Systems (Long Beach, CA, USA, December 4 - 7, 2017), Vol. 30. 1709–1720

Dan Alistarh, Demjan Grubic, Jerry Li, Ryota Tomioka, and Milan Vojnovic. 2017 · 2017

Earlier work this paper cites.

Scaling Deep Learning Workloads: NVIDIA DGX-1/Pascal and Intel Knights Landing

Nitin A. Gawande, Joshua B. Landwehr, Jeff A. Daily, Nathan R. Tallent, Abhinav Vishnu, and Darren J. Kerbyson. 2017 · 2017

Earlier work this paper cites.

Accurate, large minibatch sgd: Training imagenet in 1 hour

Original

Priya Goyal, Piotr Dollár, Ross Girshick, Pieter Noordhuis, Lukasz Wesolowski, Aapo Kyrola, Andrew Tulloch, Yangqing Jia, and Kaiming He. 2017 · 2017

Earlier work this paper cites.

Deep gradient compression: Reducing the communication bandwidth for distributed training

Original

Yujun Lin, Song Han, Huizi Mao, Yu Wang, and William J Dally. 2017 · 2017

Earlier work this paper cites.

Attention is all you need. In Advances in neural information processing systems . 5998–6008

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017 · 2017

Earlier work this paper cites.

Terngrad: Ternary gradients to reduce communication in distributed deep learning

Original

Wei Wen, Cong Xu, Feng Yan, Chunpeng Wu, Yandan Wang, Yiran Chen, and Hai Li. 2017 · 2017

Earlier work this paper cites.

ADaComP: Adaptive residual gradient compression for data-parallel distributed training. In 32nd AAAI Conference on Artificial Intelligence, AAAI 2018 (New Orleans, USA, February 2–7, 2018). 2827–2835

Chia Yu Chen, Jungwook Choi, Daniel Brand, Ankur Agrawal, Wei Zhang, and Kailash Gopalakrishnan. 2018 · 2018

Earlier work this paper cites.

Bert: Pre-training of deep bidirectional transformers for language understanding

Original

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018 · 2018

Cited alongside, same era.

Synchronous multi-gpu deep learning with low-precision communication: An experimental study. In Proceedings of the 21st International Conference on Extending Database Technology (Vienna, Austria, March 26-29, 2018). OpenProceedings, 145–156

Demjan Grubic, Leo K Tam, Dan Alistarh, and Ce Zhang. 2018 · 2018

Cited alongside, same era.

Tartan: Evaluating Modern GPU Interconnect via a Multi-GPU Benchmark Suite. In 2018 IEEE International Symposium on Workload Characterization (IISWC) (Raleigh, NC, USA, 30 September - 02 October 2018). 191–202

Ang Li, Shuaiwen Leon Song, Jieyang Chen, Xu Liu, Nathan Tallent, and Kevin Barker. 2018 · 2018

Cited alongside, same era.

3lc: Lightweight and effective traffic compression for distributed machine learning

Original

Hyeontaek Lim, David G Andersen, and Michael Kaminsky. 2018 · 2018

A Unified Architecture for Accelerating Distributed DNN Training in Heterogeneous GPU/CPU Clusters. In 14th USENIX Symposium on Operating Systems Design and Implementation (OSDI 20) (Virtual Event, November 4–6, 2020). USENIX Association, 463–479

Yimin Jiang, Yibo Zhu, Chang Lan, Bairen Yi, Yong Cui, and Chuanxiong Guo. 2020 · 2020

Later among the works it cites.

MLPerf: An industry standard benchmark suite for machine learning performance

Peter Mattson, Vijay Janapa Reddi, Christine Cheng, Cody Coleman, Greg Diamos, David Kanter, Paulius Micikevicius, David Patterson, Guenther Schmuelling, Hanlin Tang, et al · 2020

Later among the works it cites.

GRACE: A Compressed Communication Framework for Distributed Machine Learning. In Proceedings of ICDCS’21 (Virtual event, July 7- 10, 2020)

Hang Xu, Chen-Yu Ho, Ahmed M. Abdelmoniem, Aritra Dutta, El Houcine Bergou, Konstantinos Karatsenidis, Marco Canini, and Panos Kalnis. 2021 · 2020

Later among the works it cites.

Fast Training of Deep Learning Models over Multiple GPUs. In Proceedings of the 21st International Middleware Conference (Delft, Netherlands, December 7 - 11, 2020). 105–118

Xiaodong Yi, Ziyue Luo, Chen Meng, Mengdi Wang, Guoping Long, Chuan Wu, Jun Yang, and Wei Lin. 2020 · 2020