Fetching the paper…
Reading the bibliography…
We present GSPMD, an automatic, compiler-based parallelization system for common machine learning computations.
MPI: A Message-Passing Interface Standard. Version 2.2, September 4th 2009
MPI Forum · 2009
Earlier work this paper cites.
ImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., and Hinton, G. E · 2012
Earlier work this paper cites.
Adam: a Method for Stochastic Optimization
Kingma, D. P., and Ba, J. L · 2015
Earlier work this paper cites.
TensorFlow: A System for Large-Scale Machine Learning
Abadi, M., Barham, P., Chen, J., Chen, Z., Davis, A., Dean, J., Devin, M., Ghemawat, S., Irving, G., Isard, M., Kudlur, M., Levenberg, J., Monga, R., Moore, S., Murray, D. G., Steiner, B., Tucker, P., Vasudevan, V., Warden, P., Wicke, M., Yu, Y., and Zheng, X · 2016
Earlier work this paper cites.
Training deep nets with sublinear memory cost, 2016
Chen, T., Xu, B., Zhang, C., and Guestrin, C · 2016
Earlier work this paper cites.
3D U-Net: Learning dense volumetric segmentation from sparse annotation
Çiçek, Ö., Abdulkadir, A., Lienkamp, S. S., Brox, T., and Ronneberger, O · 2016
Earlier work this paper cites.
Julia: A fresh approach to numerical computing
Bezanson, J., Edelman, A., Karpinski, S., and Shah, V. B · 2017
Earlier work this paper cites.
Outrageously large neural networks: The sparsely-gated mixture-of-experts layer
Shazeer, N., Mirhoseini, A., Maziarz, K., Davis, A., Le, Q., Hinton, G., and Dean, J · 2017
Earlier work this paper cites.
Attention Is All You Need
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., and Polosukhin, I · 2017
Earlier work this paper cites.
TVM: An automated end-to-end optimizing compiler for deep learning
Chen, T., Moreau, T., Jiang, Z., Zheng, L., Yan, E., Cowan, M., Shen, H., Wang, L., Hu, Y., Ceze, L., Guestrin, C., and Krishnamurthy, A · 2018
Earlier work this paper cites.
Gpipe: Efficient training of giant neural networks using pipeline parallelism
Huang, Y., Cheng, Y., Chen, D., Lee, H., Ngiam, J., Le, Q. V., and Chen, Z · 2018
Earlier work this paper cites.
Glow: Graph lowering compiler techniques for neural networks, 2018
Rotem, N., Fix, J., Abdulrasool, S., Catron, G., Deng, S., Dzhabarov, R., Gibson, N., Hegeman, J., Lele, M., Levenstein, R., Montgomery, J., Maher, B., Nadathur, S., Olesen, J., Park, J., Rakhov, A., Smelyanskiy, M., and Wang, M · 2018
Earlier work this paper cites.
Mesh-tensorflow: Deep learning for supercomputers
Shazeer, N., Cheng, Y., Parmar, N., Tran, D., Vaswani, A., Koanantakool, P., Hawkins, P., Lee, H., Hong, M., Young, C., et al · 2018
Earlier work this paper cites.
Adafactor: Adaptive Learning Rates with Sublinear Memory Cost
Shazeer, N., and Stern, M · 2018
Cited alongside, same era.
Beyond Data and Model Parallelism for Deep Neural Networks
Jia, Z., Zaharia, M., and Aiken, A · 2019
Cited alongside, same era.
Beyond Data and Model Parallelism for Deep Neural Networks
Jia, Z., Zaharia, M., and Aiken, A · 2019
Cited alongside, same era.
PipeDream: Generalized pipeline parallelism for dnn training
Narayanan, D., Harlap, A., Phanishayee, A., Seshadri, V., Devanur, N. R., Ganger, G. R., Gibbons, P. B., and Zaharia, M · 2019
Cited alongside, same era.
PyTorch: An imperative style, high-performance deep learning library
Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., et al · 2019
Cited alongside, same era.
https://www.tensorflow.org/xla/operation_semantics
XLA operation semantics · 2021
Closest in time.
https://www.tensorflow.org/xla
XLA: Optimizing Compiler for TensorFlow · 2021
Closest in time.
https://www.microsoft.com/en-us/research/blog/deepspeed-extreme-scale-model-training-for-everyone/ , 2020
DeepSpeed: Extreme-scale model training for everyone · 2021
Closest in time.
Train ML models on large images and 3D volumes with spatial partitioning on Cloud TPUs
Cheng, Y., Lee, H., and Berghammer, T · 2021
Closest in time.
Glam: Efficient scaling of language models with mixture-of-experts, 2021
Du, N., Huang, Y., Dai, A. M., Tong, S., Lepikhin, D., Xu, Y., Krikun, M., Zhou, Y., Yu, A. W., Firat, O., Zoph, B., Fedus, L., Bosma, M., Zhou, Z., Wang, T., Wang, Y. E., Webster, K., Pellat, M., Robinson, K., Meier-Hellstern, K., Duke, T., Dixon, L., Zhang, K., Le, Q. V., Wu, Y., Chen, Z., and Cui, C · 2021
Closest in time.
alphaXiv searches the wider corpus for related work and actual follow-ups.
alphaXiv is searching for related work…
Rajbhandari, S., Rasley, J., Ruwase, O., and He, Y · 2019
Cited alongside, same era.
Megatron-LM: Training multi-billion parameter language models using GPU model parallelism
Shoeybi, M., Patwary, M., Puri, R., LeGresley, P., Casper, J., and Catanzaro, B · 2019
Cited alongside, same era.
Supporting very large models using automatic dataflow graph partitioning
Wang, M., Huang, C.-c., and Li, J · 2019
Cited alongside, same era.
Language models are few-shot learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al · 2020
Cited alongside, same era.
Conformer: Convolution-augmented transformer for speech recognition, 2020
Gulati, A., Qin, J., Chiu, C.-C., Parmar, N., Zhang, Y., Yu, J., Han, W., Wang, S., Zhang, Z., Wu, Y., and Pang, R · 2020
Cited alongside, same era.
GShard: Scaling giant models with conditional computation and automatic sharding
Lepikhin, D., Lee, H., Xu, Y., Chen, D., Firat, O., Huang, Y., Krikun, M., Shazeer, N., and Chen, Z · 2020
Cited alongside, same era.
Efficient algorithms for device placement of dnn graph operators, 2020
Tarnawski, J., Phanishayee, A., Devanur, N. R., Mahajan, D., and Paravecino, F. N · 2020
Cited alongside, same era.
Espeholt, L., Agrawal, S., Sønderby, C., Kumar, M., Heek, J., Bromberg, C., Gazen, C., Hickey, J., Bell, A., and Kalchbrenner, N · 2021
Closest in time.
DAPPLE: A pipelined data parallel approach for training large models
Fan, S., Rong, Y., Meng, C., Cao, Z., Wang, S., Zheng, Z., Wu, C., Long, G., Yang, J., Xia, L., Diao, L., Liu, X., and Lin, W · 2021
Closest in time.
Cloud TPU
Google Cloud · 2021
Closest in time.
GShard: Scaling giant models with conditional computation and automatic sharding
Lepikhin, D., Lee, H., Xu, Y., Chen, D., Firat, O., Huang, Y., Krikun, M., Shazeer, N., and Chen, Z · 2021
Closest in time.
TeraPipe: Token-level pipeline parallelism for training large-scale language models, 2021
Li, Z., Zhuang, S., Guo, S., Zhuo, D., Zhang, H., Song, D., and Stoica, I · 2021
Closest in time.
Efficient large-scale language model training on gpu clusters, 2021
Narayanan, D., Shoeybi, M., Casper, J., LeGresley, P., Patwary, M., Korthikanti, V., Vainbrand, D., Kashinkunti, P., Bernauer, J., Catanzaro, B., Phanishayee, A., and Zaharia, M · 2021
Closest in time.
Zero-shot text-to-image generation
Ramesh, A., Pavlov, M., Goh, G., Gray, S., Voss, C., Radford, A., Chen, M., and Sutskever, I · 2021
Closest in time.
Bigssl: Exploring the frontier of large-scale semi-supervised learning for automatic speech recognition, 2021
Zhang, Y., Park, D. S., Han, W., Qin, J., Gulati, A., Shor, J., Jansen, A., Xu, Y., Huang, Y., Wang, S., Zhou, Z., Li, B., Ma, M., Chan, W., Yu, J., Wang, Y., Cao, L., Sim, K. C., Ramabhadran, B., Sainath, T. N., Beaufays, F., Chen, Z., Le, Q. V., Chiu, C.-C., Pang, R., and Wu, Y · 2021
Closest in time.