Built on

G. K. Zipf, “The Psycho-Biology of Language,” Linguistic Society of America , vol. 12, no. 3, pp. 196–210, 1936
1936
Earlier work this paper cites.
G. K. Zipf, Human Behaviour and the Principle of Least Effort: an Introduction to Human Ecology . Addison-Wesley, 1949
1949
Earlier work this paper cites.
K. W. Church and R. L. Mercer, “Introduction to the Special Issue on Computational Linguistics Using Large Corpora,” Computational Linguistics , vol. 19, no. 1, pp. 1–24, Mar. 1993. [Online]. Available: http://dl.acm.org/citation.cfm?id=972450.972452
1993
Earlier work this paper cites.
M. Banko and E. Brill, “Scaling to very very large corpora for natural language disambiguation,” in 39th annual meeting on association for computational linguistics , 2001, pp. 26–33
2001
Earlier work this paper cites.
Y. Bengio, R. Ducharme, P. Vincent, and C. Jauvin, “A neural probabilistic language model,” Journal of machine learning research , vol. 3, no. Feb, pp. 1137–1155, 2003
2003
Earlier work this paper cites.
S. Bird, E. Klein, and E. Loper, Natural Language Processing with Python , 1st ed. O’Reilly Media, Inc., 2009
2009
Earlier work this paper cites.
J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei, “ImageNet: A Large-Scale Hierarchical Image Database,” in CVPR09 , 2009
2009
Earlier work this paper cites.
C. Chelba, T. Mikolov, M. Schuster, Q. Ge, T. Brants, P. Koehn, and T. Robinson, “One billion word benchmark for measuring progress in statistical language modeling,” arXiv preprint arXiv:1312.3005 , 2013
Original
2013
Earlier work this paper cites.
C. Buck, K. Heafield, and B. van Ooyen, “N-gram Counts and Language Models from the Common Crawl,” in Proceedings of the Language Resources and Evaluation Conference , Reykjavik, Iceland, May 2014. [Online]. Available: http://commoncrawl.org/
2014
Earlier work this paper cites.
J. McAuley, C. Targett, Q. Shi, and A. van den Hengel, “Image-Based Recommendations on Styles and Substitutes,” ser. SIGIR ’15. New York, NY, USA: ACM, 2015, pp. 43–52
2015
Earlier work this paper cites.
M. Luong, H. Pham, and C. D. Manning, “Effective Approaches to Attention-based Neural Machine Translation,” CoRR , vol. abs/1508.04025, 2015. [Online]. Available: http://arxiv.org/abs/1508.04025
Original
2015
Earlier work this paper cites.
D. Amodei and et. al., “Deep Speech 2: End-to-End Speech Recognition in English and Mandarin,” CoRR , vol. abs/1512.02595, 2015
Original
2015
Earlier work this paper cites.
A. M. Rush, S. Chopra, and J. Weston, “A Neural Attention Model for Abstractive Sentence Summarization,” CoRR , vol. abs/1509.00685, 2015
Original
2015
Earlier work this paper cites.

Similar

W. Chen, D. Grangier, and M. Auli, “Strategies for training large vocabulary neural language models,” arXiv:1512.04906 , 2015
Original
2015
Cited alongside, same era.
M. Abadi et al. , “TensorFlow: Large-scale machine learning on heterogeneous systems,” 2015, software available from tensorflow.org. [Online]. Available: https://www.tensorflow.org/
2015
Cited alongside, same era.
M.-S. Isabel, F.-C. Francesc, and C. Alvaro, “Large-Scale Analysis of Zipf’s Law in English Texts,” PLOS ONE , 2016
2016
Cited alongside, same era.
R. Józefowicz, O. Vinyals, M. Schuster, N. Shazeer, and Y. Wu, “Exploring the Limits of Language Modeling,” vol. abs/1602.02410, 2016
Original
2016
Cited alongside, same era.
J. Chen, X. Pan, R. Monga, S. Bengio, and R. Jozefowicz, “Revisiting distributed synchronous SGD,” arXiv preprint arXiv:1604.00981 , 2016

Then

R. Kim, “Flashblade – Now 5X Bigger, 5X Faster,” https://blog.purestorage.com/flashblade-now-5x-bigger-5x-faster , 2017
2017
Later among the works it cites.
P. Micikevicius, S. Narang, J. Alben, G. Diamos, E. Elsen, D. Garcia, B. Ginsburg, M. Houston, O. Kuchaev, G. Venkatesh et al. , “Mixed precision training,” arXiv preprint arXiv:1710.03740 , 2017
Original
2017
Later among the works it cites.
J. Hestness, S. Narang, N. Ardalani, G. F. Diamos, H. Jun, H. Kianinejad, M. M. A. Patwary, Y. Yang, and Y. Zhou, “Deep Learning Scaling is Predictable, Empirically,” CoRR , vol. abs/1712.00409, 2017
Original
2017
Later among the works it cites.
Wikipedia, “Zipfś law,” https://en.wikipedia.org/wiki/Zipf%27s_law , (Accessed on 10/12/2018)
2018
Closest in time.
S. Yu, C. Xu, and H. Liu, “Zipf’s law in 50 languages: its structural pattern, linguistic interpretation, and cognitive motivation,” arXiv:1807.01855 , 2018

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…

2018

Language Modeling at Scale

Patwary, Mostofa, Chabbi, Milind, Jun, Heewoo et al.

Understand

We show how Zipf's Law can be used to scale up language modeling (LM) to take advantage of more training data and more GPUs.

LM plays a key role in many important natural language applications such as speech recognition and machine translation.
Scaling up LM is important since it is widely accepted by the community that there is no data like more data.
Eventually, we would like to train on terabytes (TBs) of text (trillions of words).

Built on

G. K. Zipf, “The Psycho-Biology of Language,” Linguistic Society of America , vol. 12, no. 3, pp. 196–210, 1936
1936
Earlier work this paper cites.
G. K. Zipf, Human Behaviour and the Principle of Least Effort: an Introduction to Human Ecology . Addison-Wesley, 1949
1949
Earlier work this paper cites.
K. W. Church and R. L. Mercer, “Introduction to the Special Issue on Computational Linguistics Using Large Corpora,” Computational Linguistics , vol. 19, no. 1, pp. 1–24, Mar. 1993. [Online]. Available: http://dl.acm.org/citation.cfm?id=972450.972452
1993
Earlier work this paper cites.
M. Banko and E. Brill, “Scaling to very very large corpora for natural language disambiguation,” in 39th annual meeting on association for computational linguistics , 2001, pp. 26–33
2001
Earlier work this paper cites.
Y. Bengio, R. Ducharme, P. Vincent, and C. Jauvin, “A neural probabilistic language model,” Journal of machine learning research , vol. 3, no. Feb, pp. 1137–1155, 2003
2003
Earlier work this paper cites.
S. Bird, E. Klein, and E. Loper, Natural Language Processing with Python , 1st ed. O’Reilly Media, Inc., 2009
2009
Earlier work this paper cites.
J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei, “ImageNet: A Large-Scale Hierarchical Image Database,” in CVPR09 , 2009
2009
Earlier work this paper cites.
C. Chelba, T. Mikolov, M. Schuster, Q. Ge, T. Brants, P. Koehn, and T. Robinson, “One billion word benchmark for measuring progress in statistical language modeling,” arXiv preprint arXiv:1312.3005 , 2013
Original
2013
Earlier work this paper cites.
C. Buck, K. Heafield, and B. van Ooyen, “N-gram Counts and Language Models from the Common Crawl,” in Proceedings of the Language Resources and Evaluation Conference , Reykjavik, Iceland, May 2014. [Online]. Available: http://commoncrawl.org/
2014
Earlier work this paper cites.
J. McAuley, C. Targett, Q. Shi, and A. van den Hengel, “Image-Based Recommendations on Styles and Substitutes,” ser. SIGIR ’15. New York, NY, USA: ACM, 2015, pp. 43–52
2015
Earlier work this paper cites.
M. Luong, H. Pham, and C. D. Manning, “Effective Approaches to Attention-based Neural Machine Translation,” CoRR , vol. abs/1508.04025, 2015. [Online]. Available: http://arxiv.org/abs/1508.04025
Original
2015
Earlier work this paper cites.
D. Amodei and et. al., “Deep Speech 2: End-to-End Speech Recognition in English and Mandarin,” CoRR , vol. abs/1512.02595, 2015
Original
2015
Earlier work this paper cites.
A. M. Rush, S. Chopra, and J. Weston, “A Neural Attention Model for Abstractive Sentence Summarization,” CoRR , vol. abs/1509.00685, 2015
Original
2015
Earlier work this paper cites.

Similar

W. Chen, D. Grangier, and M. Auli, “Strategies for training large vocabulary neural language models,” arXiv:1512.04906 , 2015
Original
2015
Cited alongside, same era.
M. Abadi et al. , “TensorFlow: Large-scale machine learning on heterogeneous systems,” 2015, software available from tensorflow.org. [Online]. Available: https://www.tensorflow.org/
2015
Cited alongside, same era.
M.-S. Isabel, F.-C. Francesc, and C. Alvaro, “Large-Scale Analysis of Zipf’s Law in English Texts,” PLOS ONE , 2016
2016
Cited alongside, same era.
R. Józefowicz, O. Vinyals, M. Schuster, N. Shazeer, and Y. Wu, “Exploring the Limits of Language Modeling,” vol. abs/1602.02410, 2016
Original
2016
Cited alongside, same era.
J. Chen, X. Pan, R. Monga, S. Bengio, and R. Jozefowicz, “Revisiting distributed synchronous SGD,” arXiv preprint arXiv:1604.00981 , 2016

Then

R. Kim, “Flashblade – Now 5X Bigger, 5X Faster,” https://blog.purestorage.com/flashblade-now-5x-bigger-5x-faster , 2017
2017
Later among the works it cites.
P. Micikevicius, S. Narang, J. Alben, G. Diamos, E. Elsen, D. Garcia, B. Ginsburg, M. Houston, O. Kuchaev, G. Venkatesh et al. , “Mixed precision training,” arXiv preprint arXiv:1710.03740 , 2017
Original
2017
Later among the works it cites.
J. Hestness, S. Narang, N. Ardalani, G. F. Diamos, H. Jun, H. Kianinejad, M. M. A. Patwary, Y. Yang, and Y. Zhou, “Deep Learning Scaling is Predictable, Empirically,” CoRR , vol. abs/1712.00409, 2017
Original
2017
Later among the works it cites.
Wikipedia, “Zipfś law,” https://en.wikipedia.org/wiki/Zipf%27s_law , (Accessed on 10/12/2018)
2018
Closest in time.
S. Yu, C. Xu, and H. Liu, “Zipf’s law in 50 languages: its structural pattern, linguistic interpretation, and cognitive motivation,” arXiv:1807.01855 , 2018

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Original

2016

Cited alongside, same era.

I. Mitliagkas, C. Zhang, S. Hadjis, and C. Ré, “Asynchrony begets momentum, with an application to deep learning,” in Communication, Control, and Computing (Allerton) . IEEE, 2016, pp. 997–1004

2016

Cited alongside, same era.

P. H. Jin, Q. Yuan, F. Iandola, and K. Keutzer, “How to scale distributed deep learning?” arXiv preprint arXiv:1611.04581 , 2016

Original

2016

Cited alongside, same era.

S. Merity, B. McCann, and R. Socher, “Revisiting Activation Regularization for Language RNNs,” CoRR , vol. abs/1708.01009, 2017

Original

2017

Cited alongside, same era.

T. Akiba, S. Suzuki, and K. Fukuda, “Extremely Large Minibatch SGD: Training ResNet-50 on ImageNet in 15 Minutes,” CoRR , vol. abs/1711.04325, 2017

Original

2017

Cited alongside, same era.

Y. You, Z. Zhang, C. Hsieh, and J. Demmel, “100-epoch ImageNet Training with AlexNet in 24 Minutes,” CoRR , vol. abs/1709.05011, 2017

Original

2017

Cited alongside, same era.

P. Goyal, P. Dollár, R. B. Girshick, P. Noordhuis, L. Wesolowski, A. Kyrola, A. Tulloch, Y. Jia, and K. He, “Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour,” CoRR , vol. abs/1706.02677, 2017

Original

2017

Cited alongside, same era.

M. Stephen, S. K. Nitish, and S. Richard, “Regularizing and Optimizing LSTM Language Models,” CoRR , vol. abs/1708.02182, 2017

Original

2017

Cited alongside, same era.

Original

2018

Closest in time.

P. di Miceli, “Project Gutenberg,” https://www.gutenberg.org/ , 2018

2018

Closest in time.

BAIDU, “Baidu Tieba Log File,” https://tieba.baidu.com/index.html , 2018

2018

Closest in time.

M. E. Peters, M. Neumann, M. Iyyer, M. Gardner, C. Clark, K. Lee, and L. Zettlemoyer, “Deep contextualized word representations,” in Proc. of NAACL , 2018

2018

Closest in time.

A. Radford, K. Narasimhan, T. Salimans, and I. Sutskever, “Improving language understanding by generative pre-training,” https://blog.openai.com/language-unsupervised/, 2018, accessed: 2018/10/15

2018

Closest in time.

R. Puri, R. Kirby, N. Yakovenko, and B. Catanzaro, “Large Scale Language Modeling: Converging on 40GB of Text in Four Hours,” ArXiv e-prints , Aug. 2018

2018

Closest in time.

R. Anil, G. Pereyra, A. Passos, R. Ormándi, G. E. Dahl, and G. E. Hinton, “Large scale distributed neural network training through online distillation,” CoRR , vol. abs/1804.03235, 2018

Original

2018

Closest in time.

S. Merity, N. S. Keskar, and R. Socher, “An Analysis of Neural Language Modeling at Multiple Scales,” CoRR , vol. abs/1803.08240, 2018

Original

2018

Closest in time.

Y. Ueno and K. Fukuda, “Technologies behind Distributed Deep Learning: AllReduce,” https://preferredresearch.jp/2018/07/10/technologies-behind-distributed-deep-learning-allreduce/ , 2018

2018

Closest in time.