Fetching the paper…

Reading the bibliography…

2018

Large Scale Language Modeling: Converging on 40GB of Text in Four Hours

Puri, Raul, Kirby, Robert, Yakovenko, Nikolai et al.

Original
PDF
Post on X

Understand

Recent work has shown how to train Convolutional Neural Networks (CNNs) rapidly on large image datasets, then transfer the knowledge gained from these models to a variety of tasks.

Following [Radford 2017], in this work, we demonstrate similar scalability and transfer for Recurrent Neural Networks (RNNs) for Natural Language tasks.
By utilizing mixed precision arithmetic and a 32k batch size distributed across 128 NVIDIA Tesla V100 GPUs, we are able to train a character-level 4096-dimension multiplicative LSTM (mLSTM) for unsupervised text reconstruction over 3 epochs of the 40 GB Amazon Reviews dataset in four hours.
This runtime compares favorably with previous work taking one month to train the same size and configuration for one epoch over the same dataset.

Large Scale Language Modeling: Converging on 40GB of Text in Four Hours · Around

Built on

J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei, “Imagenet: A large-scale hierarchical image database,” 2009
2009
Earlier work this paper cites.
J. Turian, L. Ratinov, and Y. Bengio, “Word representations: A simple and general method for semi-supervised learning,” in Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics , ser. ACL ’10. Stroudsburg, PA, USA: Association for Computational Linguistics, 2010, pp. 384–394
2010
Earlier work this paper cites.
R. Parker, D. Graff, J. Kong, K. Chen, and K. Maeda, “English gigaword fifth edition,” 2011. [Online]. Available: https://catalog.ldc.upenn.edu/ldc2011t07
2011
Earlier work this paper cites.
F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay, “Scikit-learn: Machine learning in Python,” Journal of Machine Learning Research , vol. 12, pp. 2825–2830, 2011
2011
Earlier work this paper cites.
Y. A. LeCun, L. Bottou, G. B. Orr, and K.-R. Müller, Efficient BackProp . Berlin, Heidelberg: Springer Berlin Heidelberg, 2012, pp. 9–48. [Online]. Available: https://doi.org/10.1007/978-3-642-35289-8_3
2012
Earlier work this paper cites.
R. Pascanu, T. Mikolov, and Y. Bengio, “On the difficulty of training recurrent neural networks,” CoRR , vol. abs/1211.5063, 2012
Original
2012
Earlier work this paper cites.
J. Donahue, Y. Jia, O. Vinyals, J. Hoffman, N. Zhang, E. Tzeng, and T. Darrell, “Decaf: A deep convolutional activation feature for generic visual recognition,” International Conference on Machine Learning , 2013
2013
Earlier work this paper cites.
T. Mikolov, I. Sutskever, K. Chen, G. Corrado, and J. Dean, “Distributed representations of words and phrases and their compositionality,” CoRR , vol. abs/1310.4546, 2013
Original
2013
Earlier work this paper cites.
C. Chelba, T. Mikolov, M. Schuster, Q. Ge, T. Brants, and P. Koehn, “One billion word benchmark for measuring progress in statistical language modeling,” CoRR , vol. abs/1312.3005, 2013
Original
2013
Earlier work this paper cites.
I. Sutskever, “Training recurrent neural networks,” 2013. [Online]. Available: https://www.cs.utoronto.ca/~ilya/pubs/ilya_sutskever_phd_thesis.pdf
2013
Earlier work this paper cites.
K. Simonyan and A. Zisserman, “Very deep convolutional networks for large-scale image recognition,” CoRR , vol. abs/1409.1556, 2014
Original
2014
Earlier work this paper cites.
O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Z. Huang, A. Karpathy, A. Khosla, M. S. Bernstein, A. C. Berg, and F. Li, “Imagenet large scale visual recognition challenge,” CoRR , vol. abs/1409.0575, 2014
Original
2014
Earlier work this paper cites.
A. S. Razavian, H. Azizpour, J. Sullivan, and S. Carlsson, “CNN features off-the-shelf: an astounding baseline for recognition,” IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) , 2014
2014
Earlier work this paper cites.
J. Pennington, R. Socher, and C. D. Manning, “Glove: Global vectors for word representation,” 2014. [Online]. Available: https://www.aclweb.org/anthology/D14-1162
2014
Earlier work this paper cites.
D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” arXiv preprint arXiv:1412.6980 , 2014
Original
2014
Earlier work this paper cites.
J. McAuley, C. Targett, Q. Shi, and A. van den Hengel, “Image-based recommendations on styles and substitutes,” SIGIR , 2015
2015
Earlier work this paper cites.
K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” CoRR , vol. abs/1512.03385, 2015
Original
2015
Earlier work this paper cites.

Similar

A. M. Dai and Q. V. Le, “Semi-supervised sequence learning,” CoRR , vol. abs/1511.01432, 2015
Original
2015
Cited alongside, same era.
R. Kiros, Y. Zhu, R. Salakhutdinov, R. S. Zemel, A. Torralba, R. Urtasun, and S. Fidler, “Skip-thought vectors,” CoRR , vol. abs/1506.06726, 2015
Original
2015
Cited alongside, same era.
Y. Zhu, R. Kiros, R. S. Zemel, R. Salakhutdinov, R. Urtasun, A. Torralba, and S. Fidler, “Aligning books and movies: Towards story-like visual explanations by watching movies and reading books,” CoRR , vol. abs/1506.06724, 2015
Original
2015
Cited alongside, same era.
A. Karpathy, J. Johnson, and F. Li, “Visualizing and understanding recurrent networks,” CoRR , vol. abs/1506.02078, 2015
Original
2015
Cited alongside, same era.
B. Krause, L. Lu, I. Murray, and S. Renals, “Multiplicative LSTM for sequence modelling,” CoRR , vol. abs/1609.07959, 2016

Then

S. Gray, A. Radford, and D. P. Kingma, “Gpu kernels for block-sparse weights,” 2017. [Online]. Available: https://blog.openai.com/block-sparse-gpu-kernels/
2017
Later among the works it cites.
S. L. Smith and Q. V. Le, “A bayesian perspective on generalization and stochastic gradient descent,” CoRR , vol. abs/1710.06451, 2017
Original
2017
Later among the works it cites.
S. L. Smith, P. Kindermans, and Q. V. Le, “Don’t decay the learning rate, increase the batch size,” CoRR , vol. abs/1711.00489, 2017
Original
2017
Later among the works it cites.
E. Hoffer, I. Hubara, and D. Soudry, “Train longer, generalize better: closing the generalization gap in large batch training of neural networks,” 2017
2017
Later among the works it cites.
P. Micikevicius, S. Narang, J. Alben, G. F. Diamos, E. Elsen, D. Garcia, B. Ginsburg, M. Houston, O. Kuchaiev, G. Venkatesh, and H. Wu, “Mixed precision training,” CoRR , vol. abs/1710.03740, 2017

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Original

2016

Cited alongside, same era.

O. Melamud, J. Goldberger, and I. Dagan, “context2vec: Learning generic context embedding with bidirectional lstm,” in Proceedings of The 20th SIGNLL Conference on Computational Natural Language Learning , 01 2016, pp. 51–61

2016

Cited alongside, same era.

P. Ramachandran, P. J. Liu, and Q. V. Le, “Unsupervised pretraining for sequence to sequence learning,” CoRR , vol. abs/1611.02683, 2016

2016

Cited alongside, same era.

T. Salimans and D. P. Kingma, “Weight normalization: A simple reparameterization to accelerate training of deep neural networks,” CoRR , vol. abs/1602.07868, 2016

Original

2016

Cited alongside, same era.

T. Chen, B. Xu, C. Zhang, and C. Guestrin, “Training deep nets with sublinear memory cost,” CoRR , vol. abs/1604.06174, 2016

2016

Cited alongside, same era.

T. Akiba, S. Suzuki, and K. Fukuda, “Extremely large minibatch SGD: training resnet-50 on imagenet in 15 minutes,” CoRR , vol. abs/1711.04325, 2017

2017

Cited alongside, same era.

A. Radford, R. Józefowicz, and I. Sutskever, “Learning to generate reviews and discovering sentiment,” CoRR , vol. abs/1704.01444, 2017

2017

Cited alongside, same era.

K. He, G. Gkioxari, P. Dollár, and R. B. Girshick, “Mask R-CNN,” IEEE International Conference on Computer Vision (ICCV) , 2017

2017

Cited alongside, same era.

2017

Later among the works it cites.

A. Paszke, S. Gross, S. Chintala, G. Chanan, E. Yang, Z. DeVito, Z. Lin, A. Desmaison, L. Antiga, and A. Lerer, “Automatic differentiation in pytorch,” 2017

2017

Later among the works it cites.

A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin, “Attention is all you need,” CoRR , vol. abs/1706.03762, 2017

2017

Later among the works it cites.

S. Kornblith, J. Shlens, and Q. V. Le, “Do better imagenet models transfer better?” 2018

2018

Closest in time.

A. Radford, K. Narasimhan, T. Salimans, and I. Sutskever, “Improving language understanding by generative pre-training,” 2018. [Online]. Available: https://blog.openai.com/language-unsupervised/

2018

Closest in time.

M. Ott, S. Edunov, D. Grangier, and M. Auli, “Scaling neural machine translation,” 2018

2018

Closest in time.

M. E. Peters, M. Neumann, M. Iyyer, M. Gardner, C. Clark, K. Lee, and L. Zettlemoyer, “Deep contextualized word representations,” CoRR , vol. abs/1802.05365, 2018

2018

Closest in time.

J. Howard and S. Ruder, “Fine-tuned language models for text classification,” CoRR , vol. abs/1801.06146, 2018

2018

Closest in time.

S. Subramanian, A. Trischler, Y. Bengio, and C. J. Pal, “Learning general purpose distributed sentence representations via large scale multi-task learning,” CoRR , vol. abs/1804.00079, 2018

Original

2018

Closest in time.

D. Cer, Y. Yang, S. Kong, N. Hua, N. Limtiaco, R. S. John, N. Constant, M. Guajardo-Cespedes, S. Yuan, C. Tar, Y. Sung, B. Strope, and R. Kurzweil, “Universal sentence encoder,” CoRR , vol. abs/1803.11175, 2018

2018

Closest in time.

P. J. Liu, M. Saleh, E. Pot, B. Goodrich, R. Sepassi, L. Kaiser, and N. Shazeer, “Generating wikipedia by summarizing long sequences,” ICLR , 2018

2018

Closest in time.

NVIDIA. (2018) Mixed precision training: Choosing a scaling factor. [Online]. Available: https://docs.nvidia.com/deeplearning/sdk/mixed-precision-training/index.html#scalefactor

2018

Closest in time.