2016

Exploring the Limits of Language Modeling

Jozefowicz, Rafal, Vinyals, Oriol, Schuster, Mike et al.

Understand

In this work we explore recent advances in Recurrent Neural Networks for large scale Language Modeling, a task central to language understanding.

We extend current models to deal with two key challenges present in this task: corpora and vocabulary sizes, and complex, long term structure of language.
We perform an exhaustive study on techniques such as character Convolutional Neural Networks or Long-Short Term Memory, on the One Billion Word Benchmark.
Our best single model significantly improves state-of-the-art perplexity from 51.3 down to 30.0 (whilst reducing the number of parameters by a factor of 20), while an ensemble of models sets a new record by improving perplexity from 41.0 down to 23.7.

Built on

Handwritten digit recognition with a back-propagation network
Le Cun, B Boser, Denker, John S, Henderson, D, Howard, Richard E, Hubbard, W, and Jackel, Lawrence D · 1990
Earlier work this paper cites.
An efficient gradient-based algorithm for on-line training of recurrent network trajectories
Williams, Ronald J and Peng, Jing · 1990
Earlier work this paper cites.
Building a large annotated corpus of english: The penn treebank
Marcus, Mitchell P, Marcinkiewicz, Mary Ann, and Santorini, Beatrice · 1993
Earlier work this paper cites.
Improved backing-off for m-gram language modeling
Kneser, Reinhard and Ney, Hermann · 1995
Earlier work this paper cites.
Long short-term memory
Hochreiter, Sepp and Schmidhuber, Jürgen · 1997
Earlier work this paper cites.
Bidirectional recurrent neural networks
Schuster, Mike and Paliwal, Kuldip K · 1997
Earlier work this paper cites.
Learning to forget: Continual prediction with lstm
Gers, Felix A, Schmidhuber, Jürgen, and Cummins, Fred · 2000
Earlier work this paper cites.
Quick training of probabilistic neural nets by importance sampling
Bengio, Yoshua, Senécal, Jean-Sébastien, et al · 2003
Earlier work this paper cites.
Framewise phoneme classification with bidirectional lstm and other neural network architectures
Graves, Alex and Schmidhuber, Jürgen · 2005
Earlier work this paper cites.
Hierarchical probabilistic neural network language model
Morin, Frederic and Bengio, Yoshua · 2005
Earlier work this paper cites.
Neural probabilistic language models
Bengio, Yoshua, Schwenk, Holger, Senécal, Jean-Sébastien, Morin, Fréderic, and Gauvain, Jean-Luc · 2006
Earlier work this paper cites.
Adaptive importance sampling to accelerate training of a neural probabilistic language model
Bengio, Yoshua and Senécal, Jean-Sébastien · 2008
Earlier work this paper cites.
Imagenet: A large-scale hierarchical image database
Deng, Jia, Dong, Wei, Socher, Richard, Li, Li-Jia, Li, Kai, and Fei-Fei, Li · 2009
Earlier work this paper cites.
A scalable hierarchical distributed language model
Mnih, Andriy and Hinton, Geoffrey E · 2009
Earlier work this paper cites.
Noise-contrastive estimation: A new estimation principle for unnormalized statistical models
Gutmann, Michael and Hyvärinen, Aapo · 2010
Earlier work this paper cites.
Recurrent neural network based language model
Mikolov, Tomas, Karafiát, Martin, Burget, Lukas, Cernockỳ, Jan, and Khudanpur, Sanjeev · 2010
Earlier work this paper cites.

Similar

Empirical evaluation and combination of advanced language modeling techniques
Mikolov, Tomas, Deoras, Anoop, Kombrink, Stefan, Burget, Lukas, and Cernockỳ, Jan · 2011
Cited alongside, same era.
Generating text with recurrent neural networks
Sutskever, Ilya, Martens, James, and Hinton, Geoffrey E · 2011
Cited alongside, same era.
Deep neural network language models
Arisoy, Ebru, Sainath, Tara N, Kingsbury, Brian, and Ramabhadran, Bhuvana · 2012
Cited alongside, same era.
Imagenet classification with deep convolutional neural networks
Krizhevsky, Alex, Sutskever, Ilya, and Hinton, Geoffrey E · 2012
Cited alongside, same era.
Statistical language models based on neural networks
Mikolov, Tomáš · 2012
Cited alongside, same era.
Context dependent recurrent neural network language model
Mikolov, Tomas and Zweig, Geoffrey · 2012
Cited alongside, same era.
On the difficulty of training recurrent neural networks
Original
Pascanu, Razvan, Mikolov, Tomas, and Bengio, Yoshua · 2012
Cited alongside, same era.
Large, pruned or continuous space language models on a gpu for statistical machine translation
Schwenk, Holger, Rousseau, Anthony, and Attik, Mohammed · 2012
Cited alongside, same era.
One billion word benchmark for measuring progress in statistical language modeling
Original
Chelba, Ciprian, Mikolov, Tomas, Schuster, Mike, Ge, Qi, Brants, Thorsten, Koehn, Phillipp, and Robinson, Tony · 2013
Cited alongside, same era.
Generating sequences with recurrent neural networks
Original
Graves, Alex · 2013
Cited alongside, same era.
Learning word embeddings efficiently with noise-contrastive estimation
Mnih, Andriy and Kavukcuoglu, Koray · 2013
Cited alongside, same era.
Improving neural networks with dropout
Srivastava, Nitish · 2013
Cited alongside, same era.

Then

Sequence to sequence learning with neural networks
Sutskever, Ilya, Vinyals, Oriol, and Le, Quoc V · 2014
Later among the works it cites.
Recurrent neural network regularization
Original
Zaremba, Wojciech, Sutskever, Ilya, and Vinyals, Oriol · 2014
Later among the works it cites.
TensorFlow: Large-scale machine learning on heterogeneous systems, 2015
Abadi, Martín, Agarwal, Ashish, Barham, Paul, Brevdo, Eugene, Chen, Zhifeng, Citro, Craig, Corrado, Greg S., Davis, Andy, Dean, Jeffrey, Devin, Matthieu, Ghemawat, Sanjay, Goodfellow, Ian, Harp, Andrew, Irving, Geoffrey, Isard, Michael, Jia, Yangqing, Jozefowicz, Rafal, Kaiser, Lukasz, Kudlur, Manjunath, Levenberg, Josh, Mané, Dan, Monga, Rajat, Moore, Sherry, Murray, Derek, Olah, Chris, Schuster, Mike, Shlens, Jonathon, Steiner, Benoit, Sutskever, Ilya, Talwar, Kunal, Tucker, Paul, Vanhoucke, Vincent, Vasudevan, Vijay, Viégas, Fernanda, Vinyals, Oriol, Warden, Pete, Wattenberg, Martin, Wicke, Martin, Yu, Yuan, and Zheng, Xiaoqiang · 2015
Later among the works it cites.
Improved transition-based parsing by modeling characters instead of words with lstms
Original
Ballesteros, Miguel, Dyer, Chris, and Smith, Noah A · 2015
Later among the works it cites.
Sentence compression by deletion with lstms
Filippova, Katja, Alfonseca, Enrique, Colmenares, Carlos A, Kaiser, Lukasz, and Vinyals, Oriol · 2015
Later among the works it cites.
Multilingual language processing from bytes
Original
Gillick, Dan, Brunk, Cliff, Vinyals, Oriol, and Subramanya, Amarnag · 2015
Later among the works it cites.
An empirical exploration of recurrent network architectures
Jozefowicz, Rafal, Zaremba, Wojciech, and Sutskever, Ilya · 2015
Later among the works it cites.
Character-aware neural language models
Original
Kim, Yoon, Jernite, Yacine, Sontag, David, and Rush, Alexander M · 2015
Later among the works it cites.
Finding function in form: Compositional character models for open vocabulary word representation
Original
Ling, Wang, Luís, Tiago, Marujo, Luís, Astudillo, Ramón Fernandez, Amir, Silvio, Dyer, Chris, Black, Alan W, and Trancoso, Isabel · 2015
Later among the works it cites.
A neural attention model for abstractive sentence summarization
Original
Rush, Alexander M, Chopra, Sumit, and Weston, Jason · 2015
Later among the works it cites.
Hierarchical neural network generative models for movie dialogues
Original
Serban, Iulian Vlad, Sordoni, Alessandro, Bengio, Yoshua, Courville, Aaron C., and Pineau, Joelle · 2015
Later among the works it cites.
Sparse non-negative matrix language modeling for skip-grams
Shazeer, Noam, Pelemans, Joris, and Chelba, Ciprian · 2015
Later among the works it cites.
Efficient exact gradient update for training deep networks with very large sparse targets
Vincent, Pascal, de Brébisson, Alexandre, and Bouthillier, Xavier · 2015
Later among the works it cites.
A neural conversational model
Original
Vinyals, Oriol and Le, Quoc · 2015
Later among the works it cites.
Larger-context language modelling
Original
Wang, Tian and Cho, Kyunghyun · 2015
Later among the works it cites.
Scaling recurrent neural network language models
Williams, Will, Prasad, Niranjani, Mrva, David, Ash, Tom, and Robinson, Tony · 2015
Later among the works it cites.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…