Improving N-gram Language Models with Pre-trained Deep Transformer

“A neural probabilistic language model,”

Yoshua Bengio, Réjean Ducharme, Pascal Vincent, and Christian Jauvin, · 2003

Earlier work this paper cites.

“Recurrent neural network based language model,”

Tomáš Mikolov, Martin Karafiát, Lukáš Burget, Jan Černockỳ, and Sanjeev Khudanpur, · 2010

Earlier work this paper cites.

“A fast re-scoring strategy to capture long-distance dependencies,”

Anoop Deoras, Tomáš Mikolov, and Kenneth Church, · 2011

Earlier work this paper cites.

“Extensions of recurrent neural network language model,”

Tomáš Mikolov, Stefan Kombrink, Lukáš Burget, Jan Černockỳ, and Sanjeev Khudanpur, · 2011

Earlier work this paper cites.

“Generating text with recurrent neural networks,”

Ilya Sutskever, James Martens, and Geoffrey E Hinton, · 2011

Earlier work this paper cites.

“Lstm neural networks for language modeling,”

Martin Sundermeyer, Ralf Schlüter, and Hermann Ney, · 2012

Earlier work this paper cites.

“Converting neural network language models into back-off language models for efficient decoding in automatic speech recognition,”

Ebru Arısoy, Stanley F Chen, Bhuvana Ramabhadran, and Abhinav Sethy, · 2013

Earlier work this paper cites.

“Converting continuous-space language models into n-gram language models for statistical machine translation,”

Rui Wang, Masao Utiyama, Isao Goto, Eiichro Sumita, Hai Zhao, and Bao-Liang Lu, · 2013

Cited alongside, same era.

“Comparing approaches to convert recurrent neural networks into backoff language models for efficient decoding,”

Heike Adel, Katrin Kirchhoff, Ngoc Thang Vu, Dominic Telaar, and Tanja Schultz, · 2014

Cited alongside, same era.

“Combinations of various language model technologies including data expansion and adaptation in spontaneous speech recognition,”

Ryo Masumura, Taichi Asami, Takanobu Oba, Hirokazu Masataki, Sumitaka Sakauchi, and Akinori Ito, · 2015

Cited alongside, same era.

“Listen, attend and spell: A neural network for large vocabulary conversational speech recognition,”

William Chan, Navdeep Jaitly, Quoc Le, and Oriol Vinyals, · 2016

Cited alongside, same era.

“Neural machine translation of rare words with subword units,”

Rico Sennrich, Barry Haddow, and Alexandra Birch, · 2016

“The microsoft 2017 conversational speech recognition system,”

Wayne Xiong, Lingfeng Wu, Fil Alleva, Jasha Droppo, Xuedong Huang, and Andreas Stolcke, · 2018

Later among the works it cites.

“Improving language understanding by generative pre-training,”

Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever, · 2018

Later among the works it cites.

“Improvements to n-gram language model using text generated from neural language model,”

Masayuki Suzuki, Nobuyasu Itoh, Tohru Nagano, Gakuto Kurata, and Samuel Thomas, · 2019

Closest in time.

“Bert: Pre-training of deep bidirectional transformers for language understanding,”

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova, · 2019

Closest in time.

“Language models are unsupervised multitask learners,”

Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever, · 2019

Closest in time.

“From senones to chenones: Tied context-dependent graphemes for hybrid speech recognition,”