Fetching the paper…
Reading the bibliography…
DNN models are becoming increasingly larger to achieve unprecedented accuracy, and the accompanying increased computation and memory requirements necessitate the employment of massive clusters and elaborate parallelization strategies to accelerate DNN training.
L. G. Valiant, “A bridging model for parallel computation,” Commun. ACM , vol. 33, no. 8, pp. 103–111, 1990
1990
Earlier work this paper cites.
K. Kothapalli, R. Mukherjee, M. S. Rehman, S. Patidar, P. Narayanan, and K. Srinathan, “A performance prediction model for the cuda gpgpu platform,” in 2009 International Conference on High Performance Computing (HiPC) . IEEE, 2009, pp. 463–472
2009
Earlier work this paper cites.
Y. Zhang and J. D. Owens, “A quantitative performance analysis model for gpu architectures,” in 2011 IEEE 17th international symposium on high performance computer architecture . IEEE, 2011, pp. 382–393
2011
Earlier work this paper cites.
A. Krizhevsky, I. Sutskever, and G. E. Hinton, “Imagenet classification with deep convolutional neural networks,” Advances in neural information processing systems , vol. 25, pp. 1097–1105, 2012
2012
Earlier work this paper cites.
J. Dean, G. Corrado, R. Monga, K. Chen, M. Devin, Q. V. Le, M. Z. Mao, M. Ranzato, A. W. Senior, P. A. Tucker, K. Yang, and A. Y. Ng, “Large scale distributed deep networks,” in Advances in Neural Information Processing Systems 25: 26th Annual Conference on Neural Information Processing Systems 2012. Proceedings of a meeting held December 3-6, 2012, Lake Tahoe, Nevada, United States , P. L. Bartlett, F. C. N. Pereira, C. J. C. Burges, L. Bottou, and K. Q. Weinberger, Eds., 2012, pp. 1232–1240
2012
Earlier work this paper cites.
2014
Earlier work this paper cites.
2014
Earlier work this paper cites.
F. Yan, O. Ruwase, Y. He, and T. Chilimbi, “Performance modeling and scalability optimization of distributed deep learning systems,” in Proceedings of the 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining , 2015, pp. 1355–1364
2015
Earlier work this paper cites.
K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in Proceedings of the IEEE conference on computer vision and pattern recognition , 2016, pp. 770–778
2016
Earlier work this paper cites.
M. Abadi, P. Barham, J. Chen, Z. Chen, A. Davis, J. Dean, M. Devin, S. Ghemawat, G. Irving, M. Isard, M. Kudlur, J. Levenberg, R. Monga, S. Moore, D. G. Murray, B. Steiner, P. A. Tucker, V. Vasudevan, P. Warden, M. Wicke, Y. Yu, and X. Zheng, “Tensorflow: A system for large-scale machine learning,” in 12th USENIX Symposium on Operating Systems Design and Implementation, OSDI 2016, Savannah, GA, USA, November 2-4, 2016 . USENIX Association, 2016, pp. 265–283
2016
Earlier work this paper cites.
2016
Earlier work this paper cites.
C. Szegedy, V. Vanhoucke, S. Ioffe, J. Shlens, and Z. Wojna, “Rethinking the inception architecture for computer vision,” in Proceedings of the IEEE conference on computer vision and pattern recognition , 2016, pp. 2818–2826
2016
Earlier work this paper cites.
A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is all you need,” in Advances in neural information processing systems , 2017, pp. 5998–6008
2017
Earlier work this paper cites.
H. Qi, E. R. Sparks, and A. Talwalkar, “Paleo: A performance model for deep neural networks,” in 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings . OpenReview.net, 2017
2017
Earlier work this paper cites.
A. Radford, K. Narasimhan, T. Salimans, and I. Sutskever, “Improving language understanding by generative pre-training,” 2018
2018
Earlier work this paper cites.
T. Chen, T. Moreau, Z. Jiang, L. Zheng, E. Q. Yan, H. Shen, M. Cowan, L. Wang, Y. Hu, L. Ceze, C. Guestrin, and A. Krishnamurthy, “TVM: an automated end-to-end optimizing compiler for deep learning,” in 13th USENIX Symposium on Operating Systems Design and Implementation, OSDI 2018, Carlsbad, CA, USA, October 8-10, 2018 . USENIX Association, 2018, pp. 578–594
2018
Earlier work this paper cites.
2018
Cited alongside, same era.
A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, and I. Sutskever, “Language models are unsupervised multitask learners,” OpenAI blog , vol. 1, no. 8, p. 9, 2019
2019
Cited alongside, same era.
2019
Cited alongside, same era.
Z. Jia, M. Zaharia, and A. Aiken, “Beyond data and model parallelism for deep neural networks.” Proceedings of Machine Learning and Systems , vol. 1, pp. 1–13, 2019
2019
Cited alongside, same era.
D. Narayanan, M. Shoeybi, J. Casper, P. LeGresley, M. Patwary, V. Korthikanti, D. Vainbrand, P. Kashinkunti, J. Bernauer, B. Catanzaro, A. Phanishayee, and M. Zaharia, “Efficient large-scale language model training on GPU clusters using megatron-lm,” in SC ’21: The International Conference for High Performance Computing, Networking, Storage and Analysis, St. Louis, Missouri, USA, November 14 - 19, 2021 . ACM, 2021, pp. 58:1–58:15
2021
Later among the works it cites.
2021
Later among the works it cites.
2021
Later among the works it cites.
alphaXiv searches the wider corpus for related work and actual follow-ups.
alphaXiv is searching for related work…
M. Wang, C.-c. Huang, and J. Li, “Supporting very large models using automatic dataflow graph partitioning,” in Proceedings of the Fourteenth EuroSys Conference 2019 , 2019, pp. 1–17
2019
Cited alongside, same era.
D. Narayanan, A. Harlap, A. Phanishayee, V. Seshadri, N. R. Devanur, G. R. Ganger, P. B. Gibbons, and M. Zaharia, “Pipedream: generalized pipeline parallelism for dnn training,” in Proceedings of the 27th ACM Symposium on Operating Systems Principles , 2019, pp. 1–15
2019
Cited alongside, same era.
A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, A. Desmaison, A. Köpf, E. Z. Yang, Z. DeVito, M. Raison, A. Tejani, S. Chilamkurthy, B. Steiner, L. Fang, J. Bai, and S. Chintala, “Pytorch: An imperative style, high-performance deep learning library,” in Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada , 2019, pp. 8024–8035
2019
Cited alongside, same era.
Z. Pei, C. Li, X. Qin, X. Chen, and G. Wei, “Iteration time prediction for cnn in multi-gpu platform: modeling and analysis,” IEEE Access , vol. 7, pp. 64 788–64 797, 2019
2019
Cited alongside, same era.
Y. Huang, Y. Cheng, A. Bapna, O. Firat, D. Chen, M. X. Chen, H. Lee, J. Ngiam, Q. V. Le, Y. Wu, and Z. Chen, “Gpipe: Efficient training of giant neural networks using pipeline parallelism,” in Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada , 2019, pp. 103–112
2019
Cited alongside, same era.
Y. Ueno and R. Yokota, “Exhaustive study of hierarchical allreduce patterns for large messages between gpus,” in 19th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing, CCGRID 2019, Larnaca, Cyprus, May 14-17, 2019 . IEEE, 2019, pp. 430–439
2019
Cited alongside, same era.
2019
Cited alongside, same era.
T. B. Brown, B. Mann, N. Ryder, M. Subbiah, J. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, S. Agarwal, A. Herbert-Voss, G. Krueger, T. Henighan, R. Child, A. Ramesh, D. M. Ziegler, J. Wu, C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, S. Gray, B. Chess, J. Clark, C. Berner, S. McCandlish, A. Radford, I. Sutskever, and D. Amodei, “Language models are few-shot learners,” in Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual , H. Larochelle, M. Ranzato, R. Hadsell, M. Balcan, and H. Lin, Eds., 2020
2020
Cited alongside, same era.
2021
Later among the works it cites.
2021
Later among the works it cites.
S. Fan, Y. Rong, C. Meng, Z. Cao, S. Wang, Z. Zheng, C. Wu, G. Long, J. Yang, L. Xia, L. Diao, X. Liu, and W. Lin, “DAPPLE: a pipelined data parallel approach for training large models,” in PPoPP ’21: 26th ACM SIGPLAN Symposium on Principles and Practice of Parallel Programming, Virtual Event, Republic of Korea, February 27- March 3, 2021 . ACM, 2021, pp. 431–445
2021
Later among the works it cites.
G. Liu, S. Wang, and Y. Bao, “Seer: A time prediction model for cnns from gpu kernel’s view,” in 2021 30th International Conference on Parallel Architectures and Compilation Techniques (PACT) . IEEE, 2021, pp. 173–185
2021
Later among the works it cites.
R. Baghdadi, M. Merouani, M. Leghettas, K. Abdous, T. Arbaoui, K. Benatchba, and S. P. Amarasinghe, “A deep learning based cost model for automatic code optimization,” in Proceedings of Machine Learning and Systems, MLSys 2021, virtual, April 5-9, 2021 . mlsys.org, 2021
2021
Later among the works it cites.
V. Elango, “Pase: Parallelization strategies for efficient dnn training,” in 2021 IEEE International Parallel and Distributed Processing Symposium (IPDPS) . IEEE, 2021, pp. 1025–1034
2021
Later among the works it cites.
2021
Later among the works it cites.
“Nvidia nccl,” https://developer.nvidia.com/nccl , 2021
2021
Later among the works it cites.
Y. Ding, L. Zhu, Z. Jia, G. Pekhimenko, and S. Han, “IOS: inter-operator scheduler for CNN acceleration,” in Proceedings of Machine Learning and Systems 2021, MLSys 2021, virtual, April 5-9, 2021 , A. Smola, A. Dimakis, and I. Stoica, Eds. mlsys.org, 2021
2021
Later among the works it cites.
A. Qiao, S. K. Choe, S. J. Subramanya, W. Neiswanger, Q. Ho, H. Zhang, G. R. Ganger, and E. P. Xing, “Pollux: Co-adaptive cluster scheduling for goodput-optimized deep learning,” in 15th USENIX Symposium on Operating Systems Design and Implementation (OSDI 21) , 2021
2021
Later among the works it cites.
C. Yang, Z. Li, C. Ruan, G. Xu, C. Li, R. Chen, and F. Yan, “Perfestimator: A generic and extensible performance estimator for data parallel dnn training,” in 2021 IEEE/ACM International Workshop on Cloud Intelligence (CloudIntelligence) , 2021, pp. 13–18
2021
Later among the works it cites.
L. Zheng, Z. Li, H. Zhang, Y. Zhuang, Z. Chen, Y. Huang, Y. Wang, Y. Xu, D. Zhuo, E. P. Xing, J. E. Gonzalez, and I. Stoica, “Alpa: Automating inter- and intra-operator parallelism for distributed deep learning,” in 16th USENIX Symposium on Operating Systems Design and Implementation, OSDI 2022, Carlsbad, CA, USA, July 11-13, 2022 , M. K. Aguilera and H. Weatherspoon, Eds. USENIX Association, 2022, pp. 559–578
2022
Later among the works it cites.
C. Unger, Z. Jia, W. Wu, S. Lin, M. Baines, C. E. Q. Narvaez, V. Ramakrishnaiah, N. Prajapati, P. S. McCormick, J. Mohd-Yusof, X. Luo, D. Mudigere, J. Park, M. Smelyanskiy, and A. Aiken, “Unity: Accelerating DNN training through joint optimization of algebraic transformations and parallelization,” in 16th USENIX Symposium on Operating Systems Design and Implementation, OSDI 2022, Carlsbad, CA, USA, July 11-13, 2022 , M. K. Aguilera and H. Weatherspoon, Eds. USENIX Association, 2022, pp. 267–284
2022
Later among the works it cites.