Performance Modeling and Evaluation of Distributed Deep Learning Frameworks on GPUs

D. E. Rumelhart, G. E. Hinton, R. J. Williams et al. , “Learning representations by back-propagating errors,” Cognitive modeling , vol. 5, no. 3, p. 1, 1988

1988

Earlier work this paper cites.

J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei, “ImageNet: A large-scale hierarchical image database,” in Computer Vision and Pattern Recognition, 2009. CVPR 2009. IEEE Conference on . IEEE, 2009, pp. 248–255

2009

Earlier work this paper cites.

M. Zinkevich, M. Weimer, L. Li, and A. J. Smola, “Parallelized stochastic gradient descent,” in Advances in neural information processing systems , 2010, pp. 2595–2603

2010

Earlier work this paper cites.

C. Nvidia, “NVIDIA CUDA C programming guide,” Nvidia Corporation , vol. 120, no. 18, p. 8, 2011

2011

Earlier work this paper cites.

A. Krizhevsky, I. Sutskever, and G. E. Hinton, “ImageNet classification with deep convolutional neural networks,” in Advances in neural information processing systems , 2012, pp. 1097–1105

2012

Earlier work this paper cites.

S. Zhang, C. Zhang, Z. You, R. Zheng, and B. Xu, “Asynchronous stochastic gradient descent for DNN training,” in Acoustics, Speech and Signal Processing (ICASSP), 2013 IEEE International Conference on . IEEE, 2013, pp. 6660–6663

2013

Earlier work this paper cites.

M. Mathieu, M. Henaff, and Y. LeCun, “Fast training of convolutional networks through FFTs,” arXiv preprint arXiv:1312.5851 , 2013

Original

2013

Earlier work this paper cites.

S. Lee, J. K. Kim, X. Zheng, Q. Ho, G. A. Gibson, and E. P. Xing, “On model parallelization and scheduling strategies for distributed machine learning,” in Advances in neural information processing systems , 2014, pp. 2834–2842

2014

Earlier work this paper cites.

S. Chetlur, C. Woolley, P. Vandermersch, J. Cohen, J. Tran, B. Catanzaro, and E. Shelhamer, “cudnn: Efficient primitives for deep learning,” arXiv preprint arXiv:1410.0759 , 2014

Original

2014

Earlier work this paper cites.

M. Li, D. G. Andersen, J. W. Park, A. J. Smola, A. Ahmed, V. Josifovski, J. Long, E. J. Shekita, and B.-Y. Su, “Scaling distributed machine learning with the parameter server.” in OSDI , vol. 1, no. 10.4, 2014, p. 3

2014

Earlier work this paper cites.

Y. LeCun, Y. Bengio, and G. Hinton, “Deep learning,” Nature , vol. 521, no. 7553, pp. 436–444, 2015

2015

Cited alongside, same era.

S. Bahrampour, N. Ramakrishnan, L. Schott, and M. Shah, “Comparative study of deep learning software frameworks,” arXiv preprint arXiv:1511.06435 , 2015

Original

2015

Cited alongside, same era.

K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” arXiv preprint arXiv:1512.03385 , 2015

Original

2015

Cited alongside, same era.

C. Szegedy, W. Liu, Y. Jia, P. Sermanet, S. Reed, D. Anguelov, D. Erhan, V. Vanhoucke, and A. Rabinovich, “Going deeper with convolutions,” in Proceedings of the IEEE conference on computer vision and pattern recognition , 2015, pp. 1–9

2015

Cited alongside, same era.

J. Chen, R. Monga, S. Bengio, and R. Jozefowicz, “Revisiting distributed synchronous SGD,” arXiv preprint arXiv:1604.00981 , 2016

W. Wang and N. Srebro, “Stochastic nonconvex optimization with large minibatches,” arXiv preprint arXiv:1709.08728 , 2017

Original

2017

Closest in time.

P. Goyal, P. Dollár, R. Girshick, P. Noordhuis, L. Wesolowski, A. Kyrola, A. Tulloch, Y. Jia, and K. He, “Accurate, large minibatch SGD: Training ImageNet in 1 hour,” arXiv preprint arXiv:1706.02677 , 2017

Original

2017

Closest in time.

S. Shams, R. Platania, K. Lee, and S.-J. Park, “Evaluation of deep learning frameworks over different HPC architectures,” in Distributed Computing Systems (ICDCS), 2017 IEEE 37th International Conference on . IEEE, 2017, pp. 1389–1396

2017

Closest in time.

H. Kim, H. Nam, W. Jung, and J. Lee, “Performance analysis of CNN frameworks for GPUs,” in Performance Analysis of Systems and Software (ISPASS), 2017 IEEE International Symposium on . IEEE, 2017, pp. 55–64

2017

Closest in time.