2021

OneFlow: Redesign the Distributed Deep Learning Framework from Scratch

Yuan, Jinhui, Li, Xinqi, Cheng, Cheng et al.

Understand

Deep learning frameworks such as TensorFlow and PyTorch provide a productive interface for expressing and training a deep neural network (DNN) model on a single device or using data parallelism.

Still, they may not be flexible or efficient enough in training emerging large models on distributed devices, which require more sophisticated parallelism beyond data parallelism.
Plugins or wrappers have been developed to strengthen these frameworks for model or pipeline parallelism, but they complicate the usage and implementation of distributed deep learning.
Aiming at a simple, neat redesign of distributed deep learning frameworks for various parallelism paradigms, we present OneFlow, a novel distributed training framework based on an SBP (split, broadcast and partial-value) abstraction and the actor model.

Built on

A Universal Modular ACTOR Formalism for Artificial Intelligence
Hewitt, C., Bishop, P., and Steiger, R · 1973
Earlier work this paper cites.
Credit-based flow control for atm networks: Credit update protocol, adaptive credit allocation and statistical multiplexing
Kung, H. T., Blackwell, T., and Chapman, A · 1994
Earlier work this paper cites.
Discretized streams: Fault-tolerant streaming computation at scale
Zaharia, M., Das, T., Li, H., Hunter, T., Shenker, S., and Stoica, I · 2013
Earlier work this paper cites.
Scaling Distributed Machine Learning with the Parameter Server
Li, M., Andersen, D. G., Park, J. W., Smola, A. J., Ahmed, A., Josifovski, V., Long, J., Shekita, E. J., and Su, B.-Y · 2014
Earlier work this paper cites.
MXNet: A Flexible and Efficient Machine Learning Library for Heterogeneous Distributed System
Chen, T., Li, M., Li, Y., Lin, M., Wang, N., Wang, M., Xiao, T., Xu, B., Zhang, C., and Zhang, Z · 2015
Earlier work this paper cites.
Adam: A Method for Stochastic Optimization
Kingma, D. P. and Ba, J · 2015
Earlier work this paper cites.
TensorFlow: A System for Large-scale Machine Learning
Abadi, M., Barham, P., Chen, J., Chen, Z., Davis, A., Dean, J., Devin, M., Ghemawat, S., Irving, G., Isard, M., Kudlur, M., Levenberg, J., Monga, R., Moore, S., Murray, D. G., Steiner, B., Tucker, P., Vasudevan, V., Warden, P., Wicke, M., Yu, Y., and Zheng, X · 2016
Earlier work this paper cites.
Wide & Deep Learning for Recommender Systems
Cheng, H.-T., Koc, L., Harmsen, J., Shaked, T., Chandra, T., Aradhye, H., Anderson, G., Corrado, G., Chai, W., Ispir, M., Anil, R., Haque, Z., Hong, L., Jain, V., Liu, X., and Shah, H · 2016
Earlier work this paper cites.
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., and Sun, J · 2016
Earlier work this paper cites.
Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
Original
Goyal, P., Doll¨¢r, P., Girshick, R., Noordhuis, P., Wesolowski, L., Kyrola, A., Tulloch, A., Jia, Y., and He, K · 2017
Earlier work this paper cites.
Mobilefacenets: Efficient CNNs for Accurate Real-time Face Verification on Mobile Devices
Chen, S., Liu, Y., Gao, X., and Han, Z · 2018
Earlier work this paper cites.
Exploring Hidden Dimensions in Parallelizing Convolutional Neural Networks
Jia, Z., Lin, S., Qi, C. R., and Aiken, A · 2018
Earlier work this paper cites.
Mixed Precision Training
Micikevicius, P., Narang, S., Alben, J., Diamos, G., Elsen, E., Garcia, D., Ginsburg, B., Houston, M., Kuchaiev, O., Venkatesh, G., and Wu, H · 2018
Earlier work this paper cites.
Ray: A Distributed Framework for Emerging AI Applications
Moritz, P., Nishihara, R., Wang, S., Tumanov, A., Liaw, R., Liang, E., Paul, W., Jordan, M. I., and Stoica, I · 2018
Earlier work this paper cites.
Horovod: Fast and Easy Distributed Deep Learning in TensorFlow
Original
Sergeev, A. and Balso, M. D · 2018
Earlier work this paper cites.
Mesh-tensorflow: Deep learning for supercomputers
Shazeer, N., Cheng, Y., Parmar, N., Tran, D., Vaswani, A., Koanantakool, P., Hawkins, P., Lee, H., Hong, M., Young, C., et al · 2018
Earlier work this paper cites.

Similar

Stanza: Distributed Deep Learning with Small Communication Footprint
Original
Wu, X., Xu, H., Li, B., and Xiong, Y · 2018
Cited alongside, same era.
Demystifying Parallel and Distributed Deep Learning: An In-Depth Concurrency Analysis
Ben-Nun, T. and Hoefler, T · 2019
Cited alongside, same era.
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K · 2019
Cited alongside, same era.
TicTac: Accelerating Distributed Deep Learning with Communication Scheduling
Hashemi, S. H., Jyothi, S. A., and Campbell, R. H · 2019
Cited alongside, same era.
GPipe: Efficient Training of Giant Neural Networks using Pipeline Parallelism
Huang, Y., Cheng, Y., Bapna, A., Firat, O., Chen, D., Chen, M., Lee, H., Ngiam, J., Le, Q. V., Wu, Y., and Chen, z · 2019
Cited alongside, same era.
Beyond Data and Model Parallelism for Deep Neural Networks
Jia, Z., Zaharia, M., and Aiken, A · 2019
Cited alongside, same era.
PipeDream: Generalized Pipeline Parallelism for DNN Training
Narayanan, D., Harlap, A., Phanishayee, A., Seshadri, V., Devanur, N. R., Ganger, G. R., Gibbons, P. B., and Zaharia, M · 2019
Cited alongside, same era.
Optimizing Multi-GPU Parallelization Strategies for Deep Learning Training
Pal, S., Ebrahimi, E., Zulfiqar, A., Fu, Y., Zhang, V., Migacz, S., Nellans, D., and Gupta, P · 2019
Cited alongside, same era.
PyTorch: An Imperative Style, High-Performance Deep Learning Library
Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., Desmaison, A., Kopf, A., Yang, E., DeVito, Z., Raison, M., Tejani, A., Chilamkurthy, S., Steiner, B., Fang, L., Bai, J., and Chintala, S · 2019
Cited alongside, same era.
A Generic Communication Scheduler for Distributed DNN Training Acceleration
Peng, Y., Zhu, Y., Chen, Y., Bao, Y., Yi, B., Lan, C., Wu, C., and Guo, C · 2019
Cited alongside, same era.
Supporting Very Large Models Using Automatic Dataflow Graph Partitioning
Wang, M., Huang, C.-c., and Li, J · 2019
Cited alongside, same era.
Language Models are Few-Shot Learners
Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., Agarwal, S., Herbert-Voss, A., Krueger, G., Henighan, T., Child, R., Ramesh, A., Ziegler, D., Wu, J., Winter, C., Hesse, C., Chen, M., Sigler, E., Litwin, M., Gray, S., Chess, B., Clark, J., Berner, C., McCandlish, S., Radford, A., Sutskever, I., and Amodei, D · 2020
Cited alongside, same era.

Then

KungFu: Making Training in Distributed Machine Learning Adaptive
Mai, L., Li, G., Wagenländer, M., Fertakis, K., Brabete, A.-O., and Pietzuch, P · 2020
Later among the works it cites.
MLPerf Training Benchmark
Mattson, P., Cheng, C., Diamos, G., Coleman, C., Micikevicius, P., Patterson, D., Tang, H., Wei, G.-Y., Bailis, P., Bittorf, V., Brooks, D., Chen, D., Dutta, D., Gupta, U., Hazelwood, K., Hock, A., Huang, X., Kang, D., Kanter, D., Kumar, N., Liao, J., Narayanan, D., Oguntebi, T., Pekhimenko, G., Pentecost, L., Janapa Reddi, V., Robie, T., St John, T., Wu, C.-J., Xu, L., Young, C., and Zaharia, M · 2020
Later among the works it cites.
Merlin: A GPU Accelerated Recommendation Framework
Oldridge, E., Perez, J., Frederickson, B., Koumchatzky, N., Lee, M., Wang, Z.-H., Wu, L., Yu, F., Zamora, R., Yılmaz, O., Gunny, A. M., Nguyen, V. P., and Lee, S · 2020
Later among the works it cites.
Resource Elasticity in Distributed Deep Learning
Or, A., Zhang, H., and Freedman, M · 2020
Later among the works it cites.
ZeRO: Memory Optimizations Toward Training Trillion Parameter Models
Rajbhandari, S., Rasley, J., Ruwase, O., and He, Y · 2020
Later among the works it cites.
Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism
Original
Shoeybi, M., Patwary, M., Puri, R., LeGresley, P., Casper, J., and Catanzaro, B · 2020
Later among the works it cites.
https://github.com/microsoft/DeepSpeed , 2021
Microsoft DeepSpeed · 2021
Closest in time.
https://github.com/deepinsight/insightface , 2021
InsightFace Project · 2021
Closest in time.
https://developer.nvidia.com/nccl , 2021
NVIDIA NCCL · 2021
Closest in time.
https://developer.nvidia.com/DALI , 2021
NVIDIA Data Loading Library (DALI) · 2021
Closest in time.
Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity
Original
Fedus, W., Zoph, B., and Shazeer, N · 2021
Closest in time.
Efficient Large-Scale Language Model Training on GPU Clusters
Original
Narayanan, D., Shoeybi, M., Casper, J., LeGresley, P., Patwary, M., Korthikanti, V., Vainbrand, D., Kashinkunti, P., Bernauer, J., Catanzaro, B., Phanishayee, A., and Zaharia, M · 2021
Closest in time.
ZeRO-Infinity: Breaking the GPU Memory Wall for Extreme Scale Deep Learning
Original
Rajbhandari, S., Ruwase, O., Rasley, J., Smith, S., and He, Y · 2021
Closest in time.
Ownership: A Distributed Futures System for Fine-Grained Tasks
Wang, S., Liang, E., Oakes, E., Hindman, B., Luan, F. S., Cheng, A., and Stoica, I · 2021
Closest in time.
An efficient 2d method for training super-large deep learning models, 2021
Xu, Q., Li, S., Gong, C., and You, Y · 2021
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…