Fetching the paper…
Reading the bibliography…
Distributed training has become a pervasive and effective approach for training a large neural network (NN) model with processing massive data.
A universal modular ACTOR formalism for artificial intelligence
Carl Hewitt, Peter Bishop, and Richard Steiger · 1973
Earlier work this paper cites.
Legion: Programming Distributed Heterogeneous Architectures with Logical Regions, 2014
Michael Edward Bauer · 2014
Earlier work this paper cites.
MXNet: A Flexible and Efficient Machine Learning Library for Heterogeneous Distributed Systems, 2015
Tianqi Chen, Mu Li, Yutian Li, Min Lin, Naiyan Wang, Minjie Wang, Tianjun Xiao, Bing Xu, Chiyuan Zhang, and Zheng Zhang · 2015
Earlier work this paper cites.
TensorFlow: A system for large-scale machine learning, 2016
Martín Abadi, Paul Barham, Jianmin Chen, Zhifeng Chen, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Geoffrey Irving, Michael Isard, Manjunath Kudlur, Josh Levenberg, Rajat Monga, Sherry Moore, Derek G. Murray, Benoit Steiner, Paul Tucker, Vijay Vasudevan, Pete Warden, Martin Wicke, Yuan Yu, and Xiaoqiang Zheng · 2016
Earlier work this paper cites.
Device placement optimization with reinforcement learning
Azalia Mirhoseini, Hieu Pham, Quoc V Le, Benoit Steiner, Rasmus Larsen, Yuefeng Zhou, Naveen Kumar, Mohammad Norouzi, Samy Bengio, and Jeff Dean · 2017
Earlier work this paper cites.
Horovod: fast and easy distributed deep learning in tensorflow
Alexander Sergeev and Mike Del Balso · 2018
Earlier work this paper cites.
Mesh-tensorflow: Deep learning for supercomputers
Noam Shazeer, Youlong Cheng, Niki Parmar, Dustin Tran, Ashish Vaswani, Penporn Koanantakool, Peter Hawkins, HyoukJoong Lee, Mingsheng Hong, Cliff Young, et al · 2018
Earlier work this paper cites.
Gandiva: Introspective cluster scheduling for deep learning
Wencong Xiao, Romil Bhardwaj, Ramachandran Ramjee, Muthian Sivathanu, Nipun Kwatra, Zhenhua Han, Pratyush Patel, Xuan Peng, Hanyu Zhao, Quanlu Zhang, Fan Yang, and Lidong Zhou · 2018
Earlier work this paper cites.
GAI: A Centralized Tree-Based Scheduler for Machine Learning Workload in Large Shared Clusters
Ce Gao, Rui Ren, and Hongming Cai · 2018
Earlier work this paper cites.
Megatron-lm: Training multi-billion parameter language models using model parallelism
Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro · 2019
Earlier work this paper cites.
Deep Learning Recommendation Model for Personalization and Recommendation Systems, 2019
Maxim Naumov, Dheevatsa Mudigere, Hao-Jun Michael Shi, Jianyu Huang, Narayanan Sundaraman, Jongsoo Park, Xiaodong Wang, Udit Gupta, Carole-Jean Wu, Alisson G. Azzolini, Dmytro Dzhulgakov, Andrey Mallevich, Ilia Cherniavskii, Yinghai Lu, Raghuraman Krishnamoorthi, Ansha Yu, Volodymyr Kondratenko, Stephanie Pereira, Xianjie Chen, Wenlin Chen, Vijay Rao, Bill Jia, Liang Xiong, and Misha Smelyanskiy · 2019
Earlier work this paper cites.
GPipe: Efficient Training of Giant Neural Networks using Pipeline Parallelism, 2019
Yanping Huang, Youlong Cheng, Ankur Bapna, Orhan Firat, Mia Xu Chen, Dehao Chen, HyoukJoong Lee, Jiquan Ngiam, Quoc V. Le, Yonghui Wu, and Zhifeng Chen · 2019
Earlier work this paper cites.
PipeDream: Generalized pipeline parallelism for DNN training
Deepak Narayanan, Aaron Harlap, Amar Phanishayee, Vivek Seshadri, Nikhil R. Devanur, Gregory R. Ganger, Phillip B. Gibbons, and Matei Zaharia · 2019
Earlier work this paper cites.
Pytorch: An imperative style, high-performance deep learning library
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al · 2019
Earlier work this paper cites.
Deep Learning based Recommender System: A Survey and New Perspectives
Shuai Zhang, Lina Yao, Aixin Sun, and Yi Tay · 2019
Cited alongside, same era.
Big Transfer (BiT): General Visual Representation Learning, 2020
Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Joan Puigcerver, Jessica Yung, Sylvain Gelly, and Neil Houlsby · 2020
Cited alongside, same era.
Language Models are Few-Shot Learners, 2020
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei · 2020
Cited alongside, same era.
Distributed Hierarchical GPU Parameter Server for Massive Scale Deep Learning Ads Systems, 2020
Weijie Zhao, Deping Xie, Ronglai Jia, Yulei Qian, Ruiquan Ding, Mingming Sun, and Ping Li · 2020
Cited alongside, same era.
ZeRO: Memory Optimizations Toward Training Trillion Parameter Models, 2020
Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, and Yuxiong He · 2020
PipeTransformer: Automated Elastic Pipelining for Distributed Training of Transformers
Chaoyang He, Shen Li, Mahdi Soltanolkotabi, and Salman Avestimehr · 2021
Closest in time.
URL https://www.mindspore.cn/en
MindSpore, 2021 · 2021
Closest in time.
OneFlow: Redesign the Distributed Deep Learning Framework from Scratch
Jinhui Yuan, Xinqi Li, Cheng Cheng, Juncheng Liu, Ran Guo, Shenghang Cai, Chi Yao, Fei Yang, Xiaodong Yi, Chuan Wu, Haoran Zhang, and Jie Zhao · 2021
Closest in time.
GSPMD: General and Scalable Parallelization for ML Computation Graphs, 2021
Yuanzhong Xu, HyoukJoong Lee, Dehao Chen, Blake Hechtman, Yanping Huang, Rahul Joshi, Maxim Krikun, Dmitry Lepikhin, Andy Ly, Marcello Maggioni, Ruoming Pang, Noam Shazeer, Shibo Wang, Tao Wang, Yonghui Wu, and Zhifeng Chen · 2021
Closest in time.
Towards optimal placement and scheduling of dnn operations with pesto
Ubaid Ullah Hafeez, Xiao Sun, Anshul Gandhi, and Zhenhua Liu · 2021
Closest in time.
alphaXiv searches the wider corpus for related work and actual follow-ups.
alphaXiv is searching for related work…
Cited alongside, same era.
Gshard: Scaling giant models with conditional computation and automatic sharding
Zhifeng Chen, Dmitry Dima Lepikhin, HyoukJoong Lee, Maxim Krikun, Noam Shazeer, Orhan Firat, Yanping Huang, Yuanzhong Xu, and Zhifeng Chen · 2020
Cited alongside, same era.
Autosync: Learning to synchronize for data-parallel distributed deep learning
Hao Zhang, Yuan Li, Zhijie Deng, Xiaodan Liang, Lawrence Carin, and Eric Xing · 2020
Cited alongside, same era.
TensorOpt: Exploring the Tradeoffs in Distributed DNN Training with Auto-Parallelism, 2020
Zhenkun Cai, Kaihao Ma, Xiao Yan, Yidi Wu, Yuzhen Huang, James Cheng, Teng Su, and Fan Yu · 2020
Cited alongside, same era.
Automatic Cross-Replica Sharding of Weight Update in Data-Parallel Training, 2020
Yuanzhong Xu, HyoukJoong Lee, Dehao Chen, Hongjun Choi, Blake Hechtman, and Shibo Wang · 2020
Cited alongside, same era.
Effective elastic scaling of deep learning workloads
Vaibhav Saxena, KR Jayaram, Saurav Basu, Yogish Sabharwal, and Ashish Verma · 2020
Cited alongside, same era.
An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale, 2021
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby · 2021
Cited alongside, same era.
Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity, 2021
William Fedus, Barret Zoph, and Noam Shazeer · 2021
Cited alongside, same era.
Shiqing Fan, Yi Rong, Chen Meng, Zongyan Cao, Siyu Wang, Zhen Zheng, Chuan Wu, Guoping Long, Jun Yang, Lixue Xia, et al · 2021
Closest in time.
Pollux: Co-adaptive Cluster Scheduling for Goodput-Optimized Deep Learning, 2021
Aurick Qiao, Sang Keun Choe, Suhas Jayaram Subramanya, Willie Neiswanger, Qirong Ho, Hao Zhang, Gregory R. Ganger, and Eric P. Xing · 2021
Closest in time.
Ernie 3.0: Large-scale knowledge enhanced pre-training for language understanding and generation, 2021
Yu Sun, Shuohuan Wang, Shikun Feng, Siyu Ding, Chao Pang, Junyuan Shang, Jiaxiang Liu, Xuyi Chen, Yanbin Zhao, Yuxiang Lu, Weixin Liu, Zhihua Wu, Weibao Gong, Jianzhong Liang, Zhizhou Shang, Peng Sun, Wei Liu, Xuan Ouyang, Dianhai Yu, Hao Tian, Hua Wu, and Haifeng Wang · 2021
Closest in time.
URL https://github.com/deepinsight/insightface/tree/master/recognition/arcface_paddle
PaddlePaddle, 2021 · 2021
Closest in time.
ZeRO-Infinity: Breaking the GPU Memory Wall for Extreme Scale Deep Learning
Samyam Rajbhandari, Olatunji Ruwase, Jeff Rasley, Shaden Smith, and Yuxiong He · 2021
Closest in time.
Elastic deep learning in multi-tenant gpu clusters
Yidi Wu, Kaihao Ma, Xiao Yan, Zhi Liu, Zhenkun Cai, Yuzhen Huang, James Cheng, Han Yuan, and Fan Yu · 2021
Closest in time.
Towards heterogeneous clients with elastic federated learning
Zichen Ma, Yu Lu, Zihan Lu, Wenye Li, Jinfeng Yi, and Shuguang Cui · 2021
Closest in time.
An optimal resource allocator of elastic training for deep learning jobs on cloud
Liang Hu, Jiangcheng Zhu, Zirui Zhou, Ruiqing Cheng, Xiaolong Bai, and Yong Zhang · 2021
Closest in time.