2019

XPipe: Efficient Pipeline Model Parallelism for Multi-GPU DNN Training

Guan, Lei, Yin, Wotao, Li, Dongsheng et al.

Understand

We propose XPipe, an efficient asynchronous pipeline model parallelism approach for multi-GPU DNN training.

XPipe is designed to use multiple GPUs to concurrently and continuously train different parts of a DNN model.
To improve GPU utilization and achieve high throughput, it splits a mini-batch into a set of micro-batches.
It allows the overlapping of the pipelines of multiple micro-batches, including those belonging to different mini-batches.

Built on

A. Petrowski, G. Dreyfus, and C. Girault, “Performance analysis of a pipelined backpropagation parallel algorithm,” IEEE Transactions on Neural Networks , vol. 4, no. 6, pp. 970–981, 1993
1993
Earlier work this paper cites.
N. Qian, “On the momentum term in gradient descent learning algorithms,” Neural networks , vol. 12, no. 1, pp. 145–151, 1999
1999
Earlier work this paper cites.
H. Lee, P. Pham, Y. Largman, and A. Y. Ng, “Unsupervised feature learning for audio classification using convolutional deep belief networks,” in Advances in neural information processing systems , 2009, pp. 1096–1104
2009
Earlier work this paper cites.
J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei, “Imagenet: A large-scale hierarchical image database,” in 2009 IEEE conference on computer vision and pattern recognition . Ieee, 2009, pp. 248–255
2009
Earlier work this paper cites.
A. Krizhevsky and G. Hinton, “Learning multiple layers of features from tiny images,” Citeseer, Tech. Rep., 2009
2009
Earlier work this paper cites.
J. Dean, G. Corrado, R. Monga, K. Chen, M. Devin, M. Mao, M. Ranzato, A. Senior, P. Tucker, K. Yang et al. , “Large scale distributed deep networks,” in Advances in neural information processing systems , 2012, pp. 1223–1231
2012
Earlier work this paper cites.
X. Chen, A. Eversole, G. Li, D. Yu, and F. Seide, “Pipelined back-propagation for context-dependent deep neural networks,” in Thirteenth Annual Conference of the International Speech Communication Association , 2012
2012
Earlier work this paper cites.
T. Tieleman and G. Hinton, “Lecture 6.5-rmsprop: Divide the gradient by a running average of its recent magnitude,” COURSERA: Neural networks for machine learning , vol. 4, no. 2, pp. 26–31, 2012
2012
Earlier work this paper cites.
M. Kamruzzaman, S. Swanson, and D. M. Tullsen, “Load-balanced pipeline parallelism,” in SC’13: Proceedings of the International Conference on High Performance Computing, Networking, Storage and Analysis . IEEE, 2013, pp. 1–12
2013
Earlier work this paper cites.
A. Karpathy, G. Toderici, S. Shetty, T. Leung, R. Sukthankar, and L. Fei-Fei, “Large-scale video classification with convolutional neural networks,” in Proceedings of the IEEE conference on Computer Vision and Pattern Recognition , 2014, pp. 1725–1732
2014
Earlier work this paper cites.
N. Kalchbrenner, E. Grefenstette, and P. Blunsom, “A convolutional neural network for modelling sentences,” arXiv preprint arXiv:1404.2188 , 2014
Original
2014
Earlier work this paper cites.
Y. Taigman, M. Yang, M. Ranzato, and L. Wolf, “Deepface: Closing the gap to human-level performance in face verification,” in Proceedings of the IEEE conference on computer vision and pattern recognition , 2014, pp. 1701–1708
2014
Earlier work this paper cites.

Similar

S. Lee, J. K. Kim, X. Zheng, Q. Ho, G. A. Gibson, and E. P. Xing, “On model parallelization and scheduling strategies for distributed machine learning,” in Advances in neural information processing systems , 2014, pp. 2834–2842
2014
Cited alongside, same era.
D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” arXiv preprint arXiv:1412.6980 , 2014
Original
2014
Cited alongside, same era.
K. Simonyan and A. Zisserman, “Very deep convolutional networks for large-scale image recognition,” arXiv preprint arXiv:1409.1556 , 2014
Original
2014
Cited alongside, same era.
G. Hinton, O. Vinyals, and J. Dean, “Distilling the knowledge in a neural network,” arXiv preprint arXiv:1503.02531 , 2015
Original
2015
Cited alongside, same era.
L. Yao and J. Miller, “Tiny imagenet classification with convolutional neural networks,” CS 231N , vol. 2, no. 5, p. 8, 2015
2015
Cited alongside, same era.
Y. Wu, M. Schuster, Z. Chen, Q. V. Le, M. Norouzi, W. Macherey, M. Krikun, Y. Cao, Q. Gao, K. Macherey et al. , “Google’s neural machine translation system: Bridging the gap between human and machine translation,” arXiv preprint arXiv:1609.08144 , 2016
Original
2016
Cited alongside, same era.
K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in Proceedings of the IEEE conference on computer vision and pattern recognition , 2016, pp. 770–778
2016
Cited alongside, same era.
C. Szegedy, V. Vanhoucke, S. Ioffe, J. Shlens, and Z. Wojna, “Rethinking the inception architecture for computer vision,” in The IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , June 2016
2016
Cited alongside, same era.
C. Szegedy, S. Ioffe, V. Vanhoucke, and A. A. Alemi, “Inception-v4, inception-resnet and the impact of residual connections on learning,” in Thirty-First AAAI Conference on Artificial Intelligence , 2017
2017
Cited alongside, same era.
N. Shazeer, A. Mirhoseini, K. Maziarz, A. Davis, Q. Le, G. Hinton, and J. Dean, “Outrageously large neural networks: The sparsely-gated mixture-of-experts layer,” arXiv preprint arXiv:1701.06538 , 2017
Original
2017
Cited alongside, same era.
I. Krasin, T. Duerig, N. Alldrin, V. Ferrari, S. Abu-El-Haija, A. Kuznetsova, H. Rom, J. Uijlings, S. Popov, A. Veit et al. , “Openimages: A public dataset for large-scale multi-label and multi-class image classification,” Dataset available from https://github. com/openimages , vol. 2, p. 3, 2017
2017
Cited alongside, same era.
A. L. Gaunt, M. A. Johnson, M. Riechert, D. Tarlow, R. Tomioka, D. Vytiniotis, and S. Webster, “Ampnet: Asynchronous model-parallel training for dynamic neural networks,” arXiv preprint arXiv:1705.09786 , 2017
Original
2017
Cited alongside, same era.

Then

A. Paszke, S. Gross, S. Chintala, G. Chanan, E. Yang, Z. DeVito, Z. Lin, A. Desmaison, L. Antiga, and A. Lerer, “Automatic differentiation in pytorch,” 2017
2017
Later among the works it cites.
A. Mirhoseini, H. Pham, Q. V. Le, B. Steiner, R. Larsen, Y. Zhou, N. Kumar, M. Norouzi, S. Bengio, and J. Dean, “Device placement optimization with reinforcement learning,” in Proceedings of the 34th International Conference on Machine Learning-Volume 70 . JMLR. org, 2017, pp. 2430–2439
2017
Later among the works it cites.
T. Afouras, J. S. Chung, A. Senior, O. Vinyals, and A. Zisserman, “Deep audio-visual speech recognition,” IEEE transactions on pattern analysis and machine intelligence , 2018
2018
Later among the works it cites.
Y. You, Z. Zhang, C.-J. Hsieh, J. Demmel, and K. Keutzer, “Imagenet training in minutes,” in Proceedings of the 47th International Conference on Parallel Processing . ACM, 2018, p. 1
2018
Later among the works it cites.
T. Ben-Nun and T. Hoefler, “Demystifying parallel and distributed deep learning: An in-depth concurrency analysis,” arXiv preprint arXiv:1802.09941 , 2018
Original
2018
Later among the works it cites.
C.-C. Chen, C.-L. Yang, and H.-Y. Cheng, “Efficient and robust parallel dnn training through model parallelism on multi-gpu platform,” arXiv preprint arXiv:1809.02839 , 2018
Original
2018
Later among the works it cites.
Z. Huo, B. Gu, and H. Huang, “Training neural networks using features replay,” in Advances in Neural Information Processing Systems , 2018, pp. 6659–6668
2018
Later among the works it cites.
Z. Huo, B. Gu, Q. Yang, and H. Huang, “Decoupled parallel backpropagation with convergence guarantee,” arXiv preprint arXiv:1804.10574 , 2018
Original
2018
Later among the works it cites.
R. Pittman, H. Guan, X. Shen, S.-H. Lim, and R. M. Patton, “Exploring flexible communications for streamlining dnn ensemble training pipelines,” in Proceedings of the International Conference for High Performance Computing, Networking, Storage, and Analysis . IEEE Press, 2018, p. 64
2018
Later among the works it cites.
Y. Huang, Y. Cheng, A. Bapna, O. Firat, D. Chen, M. Chen, H. Lee, J. Ngiam, Q. V. Le, Y. Wu et al. , “Gpipe: Efficient training of giant neural networks using pipeline parallelism,” in Advances in Neural Information Processing Systems , 2019, pp. 103–112
2019
Closest in time.
D. Narayanan, A. Harlap, A. Phanishayee, V. Seshadri, N. R. Devanur, G. R. Ganger, P. B. Gibbons, and M. Zaharia, “Pipedream: generalized pipeline parallelism for dnn training,” in Proceedings of the 27th ACM Symposium on Operating Systems Principles . ACM, 2019, pp. 1–15
2019
Closest in time.
S. Pal, E. Ebrahimi, A. Zulfiqar, Y. Fu, V. Zhang, S. Migacz, D. Nellans, and P. Gupta, “Optimizing multi-gpu parallelization strategies for deep learning training,” IEEE Micro , vol. 39, no. 5, pp. 91–101, 2019
2019
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…