2020

Training Large Neural Networks with Constant Memory using a New Execution Algorithm

Pudipeddi, Bharadwaj, Mesmakhosroshahi, Maral, Xi, Jinwen et al.

Understand

Widely popular transformer-based NLP models such as BERT and Turing-NLG have enormous capacity trending to billions of parameters.

Current execution methods demand brute-force resources such as HBM devices and high speed interconnectivity for data parallelism.
In this paper, we introduce a new relay-style execution technique called L2L (layer-to-layer) where at any given moment, the device memory is primarily populated only with the executing layer(s)'s footprint.
The model resides in the DRAM memory attached to either a CPU or an FPGA as an entity we call eager param-server (EPS).

Large batch optimization for deep learning: Training bert in 76 minutes
Original
You, Y., Li, J., Hseu, J., Song, X., Demmel, J., and Hsieh, C. (2019) · 1904
Earlier work this paper cites.
Zero: Memory optimization towards training a trillion parameter models
Original
Rajbhandari, S., Rasley, J., Ruwase, O., and He, Y. (2019) · 1910
Earlier work this paper cites.
Huggingface’s transformers: State-of-the-art natural language processing
Original
Wolf, T., Debut, L., Sanh, V., Chaumond, J., Delangue, C., Moi, A., Cistac, P., Rault, T., Louf, R., Funtowicz, M., and Brew, J. (2019) · 1910
Earlier work this paper cites.
Language models are few-shot learners
Original
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., Agarwal, S., Herbert-Voss, A., Krueger, G., Henighan, T., Child, R., Ramesh, A., Ziegler, D. M., Wu, J., Winter, C., Hesse, C., Chen, M., Sigler, E., Litwin, M., Gray, S., Chess, B., Clark, J., Berner, C., McCandlish, S., Radford, A., Sutskever, I., and Amodei, D. (2020) · 2005
Earlier work this paper cites.
Large scale distributed deep networks
Dean, J., Corrado, G., Monga, R., Chen, K., Devin, M., Le, Q. V., Mao, M. Z., Ranzato, M., Senior, A. W., Tucker, P. A., Yang, K., and Ng, A. Y. (2012) · 2012
Earlier work this paper cites.

One weird trick for parallelizing convolutional neural networks
Original
Krizhevsky, A. (2014) · 2014
Cited alongside, same era.
Asynchronous parallel stochastic gradient for nonconvex optimization
Lian, X., Huang, Y., Li, Y., and Liu, J. (2015) · 2015
Cited alongside, same era.
vdnn: Virtualized deep neural networks for scalable, memory-efficient neural network design
Original
M. Rhu, N. Gimelshein, J. C. A. Z. S. W. K. (2016) · 2016
Cited alongside, same era.
Staleness-aware async-sgd for distributed deep learning
Zhang, W., Gupta, S., Lian, X., and Liu, J. (2016) · 2016
Cited alongside, same era.
Fitting larger networks into memory
Bulatov, Y. (2018) · 2018
Cited alongside, same era.
Revisiting distributed synchronous SGD
Original
Chen, J., Monga, R., Bengio, S., and Józefowicz, R. (2016a)
Cited in the paper.
Training deep nets with sublinear memory cost
Original
Chen, T., Xu, B., Zhang, C., and Guestrin, C. (2016b)
Cited in the paper.

Pipedream: Fast and efficient pipeline parallel dnn training
Original
Harlap, A., Narayanan, D., Phanishayee, A., Seshadri, V., Devanur, N., Ganger, G., and Gibbons, P. (2018) · 2018
Later among the works it cites.
Gpipe: Efficient training of giant neural networks using pipeline parallelism
Original
Huang, Y., Cheng, Y., Bapna, A., Firat, O., Chen, M. X., Chen, D., Lee, H., Ngiam, J., Le, Q. V., Wu, Y., and Chen, Z. (2018) · 2018
Later among the works it cites.
Bert: Pre-training of deep bidirectional transformers for language understanding
Devlin, J., Chang, M., Lee, K., and Toutanova, K. (2019) · 2019
Later among the works it cites.
Language models are unsupervised multitask learners
Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., and Sutskever, I. (2019) · 2019
Later among the works it cites.
Glue: A multi-task benchmark and analysis platform for natural language understanding
Wang, A., Singh, A., Michael, J., Hill, F., Levy, O., and Bowman1, S. R. (2019) · 2019
Later among the works it cites.

alphaXiv searches the wider corpus for related work and actual follow-ups.

alphaXiv is searching for related work…

Reading the bibliography…