Optimized Broadcast for Deep Learning Workloads on Dense-GPU InfiniBand Clusters: MPI or NCCL?

M. Barnett, L. Shuler, R. van de Geijn, S. Gupta, D. G. Payne, and J. Watts, “Interprocessor collective communication library (InterCom),” in Proceedings of IEEE Scalable High Performance Computing Conference , May 1994, pp. 357–364

1994

Earlier work this paper cites.

M. Shroff and R. A. V. D. Geijn, “CollMark: MPI Collective Communication Benchmark,” Dept. of Computer Sciences, University of Texas at Austin, Tech. Rep., 2000

2000

Earlier work this paper cites.

J. Liu, A. R. Mamidala, and D. K. Panda, “Fast and Scalable MPI-level Broadcast using InfiniBand’s Hardware Multicast Support,” in Parallel and Distributed Processing Symposium, 2004. Proceedings. 18th International , April 2004, p. 10

2004

Earlier work this paper cites.

R. Thakur, R. Rabenseifner, and W. Gropp, “Optimization of Collective Communication Operations in MPICH,” Int. J. High Perform. Comput. Appl. , vol. 19, no. 1, pp. 49–66, Feb. 2005

2005

Earlier work this paper cites.

A. R. Mamidala, L. Chai, H.-W. Jin, and D. K. Panda, “Efficient SMP-aware MPI-level Broadcast over InfiniBand’s Hardware Multicast,” in Proceedings 20th IEEE International Parallel Distributed Processing Symposium , April 2006, p. 8

2006

Earlier work this paper cites.

T. Chiba, T. Endo, and S. Matsuoka, “High-Performance MPI Broadcast Algorithm for Grid Environments Utilizing Multi-lane NICs,” in Seventh IEEE International Symposium on Cluster Computing and the Grid (CCGrid ’07) , May 2007, pp. 487–494

2007

Earlier work this paper cites.

T. Hoefler, C. Siebert, and W. Rehm, “A Practically Constant-time MPI Broadcast Algorithm for Large-scale InfiniBand Clusters with Multicast,” in Proceedings of the 21st IEEE International Parallel & Distributed Processing Symposium (CAC’07 Workshop) , Mar. 2007, p. 232

2007

Earlier work this paper cites.

J. Dean, G. Corrado, R. Monga, K. Chen, M. Devin, M. Mao, M. aurelio Ranzato, A. Senior, P. Tucker, K. Yang, Q. V. Le, and A. Y. Ng, “Large scale distributed deep networks,” in Advances in Neural Information Processing Systems 25 , F. Pereira, C. J. C. Burges, L. Bottou, and K. Q. Weinberger, Eds. Curran Associates, Inc., 2012, pp. 1223–1231. [Online]. Available: http://papers.nips.cc/paper/4687-large-scale-distributed-deep-networks.pdf

2012

Earlier work this paper cites.

T. Hoefler, J. Dinan, D. Buntinas, P. Balaji, B. W. Barrett, R. Brightwell, W. Gropp, V. Kale, and R. Thakur, “Leveraging MPI’s One-sided Communication Interface for Shared-memory Programming,” in Proceedings of the 19th European Conference on Recent Advances in the Message Passing Interface , ser. EuroMPI’12. Berlin, Heidelberg: Springer-Verlag, 2012, pp. 132–141

2012

Earlier work this paper cites.

K. Kandalla, A. Venkatesh, K. Hamidouche, S. Potluri, D. Bureddy, and D. K. Panda, “Designing Optimized MPI Broadcast and Allreduce for Many Integrated Core (MIC) InfiniBand Clusters,” in 2013 IEEE 21st Annual Symposium on High-Performance Interconnects , Aug 2013, pp. 63–70

2013

Earlier work this paper cites.

S. Potluri, K. Hamidouche, A. Venkatesh, D. Bureddy, and D. K. Panda, “Efficient Inter-node MPI Communication Using GPUDirect RDMA for InfiniBand Clusters with NVIDIA GPUs,” in Parallel Processing (ICPP), 2013 42nd International Conference on , Oct 2013, pp. 80–89

2013

Earlier work this paper cites.

Y. Jia, E. Shelhamer, J. Donahue, S. Karayev, J. Long, R. Girshick, S. Guadarrama, and T. Darrell, “Caffe: Convolutional Architecture for Fast Feature Embedding,” arXiv preprint arXiv:1408.5093 , 2014

Original

2014

Cited alongside, same era.

R. Shi, S. Potluri, K. Hamidouche, J. Perkins, M. Li, D. Rossetti, and D. K. Panda, “Designing Efficient Small Message Transfer Mechanism for Inter-node MPI Communication on InfiniBand GPU Clusters,” in 2014 21st International Conference on High Performance Computing (HiPC) , Dec 2014, pp. 1–10

2014

Cited alongside, same era.

K. Simonyan and A. Zisserman, “Very Deep Convolutional Networks for Large-Scale Image Recognition,” arXiv preprint arXiv:1409.1556 , 2014

Original

2014

Cited alongside, same era.

A. Venkatesh, H. Subramoni, K. Hamidouche, and D. K. Panda, “A High Performance Broadcast Design with Hardware Multicast and GPUDirect RDMA for Streaming Applications on Infiniband Clusters,” in 2014 21st International Conference on High Performance Computing (HiPC) , Dec 2014, pp. 1–10

A. A. Awan, K. Hamidouche, A. Venkatesh, and D. K. Panda, “Efficient Large Message Broadcast using NCCL and CUDA-Aware MPI for Deep Learning,” in Proceedings of the 23rd European MPI Users’ Group Meeting . ACM, 2016, pp. 15–22

2016

Later among the works it cites.

D. S. Banerjee, K. Hamidouche, and D. K. Panda, “Re-Designing CNTK Deep Learning Framework on Modern GPU Enabled Clusters,” in 2016 IEEE International Conference on Cloud Computing Technology and Science (CloudCom) , Dec 2016, pp. 144–151

2016

Later among the works it cites.

R. Sennrich, B. Haddow, and A. Birch, “Edinburgh Neural Machine Translation Systems for WMT 16,” CoRR , vol. abs/1606.02891, 2016. [Online]. Available: http://arxiv.org/abs/1606.02891

Original

2016

Later among the works it cites.

A. A. Awan, K. Hamidouche, J. M. Hashmi, and D. K. Panda, “S-Caffe: Co-designing MPI Runtimes and Caffe for Scalable Deep Learning on Modern GPU Clusters,” in Proceedings of the 22Nd ACM SIGPLAN Symposium on Principles and Practice of Parallel Programming , ser. PPoPP ’17. ACM, 2017, pp. 193–205