Fetching the paper…
Reading the bibliography…
Large-scale language models (LLMs) such as GPT-2, BERT and RoBERTa have been successfully applied to ASR N-best rescoring.
R. Rosenfeld, “Two decades of statistical language modeling: where do we go from here?” in Proceedings of the IEEE , 2000
2000
Earlier work this paper cites.
T. Mikolov, M. Karafiát, L. Burget, J. H. Cernocký, and S. Khudanpur, “Recurrent neural network based language model,” in INTERSPEECH , 2010
2010
Earlier work this paper cites.
L. Wan, M. Zeiler, S. Zhang, Y. Le Cun, and R. Fergus, “Regularization of neural networks using dropconnect,” in ICML , 2013
2013
Earlier work this paper cites.
E. Arisoy, A. Sethy, B. Ramabhadran, and S. Chen, “Bidirectional recurrent neural network language models for automatic speech recognition,” in ICASSP , 2015
2015
Earlier work this paper cites.
T. Ko, V. Peddinti, D. Povey, and S. Khudanpur, “Audio augmentation for speech recognition,” in INTERSPEECH , 2015
2015
Earlier work this paper cites.
2015
Earlier work this paper cites.
A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is all you need,” in NeurIPS , 2017
2017
Earlier work this paper cites.
G. Saon, G. Kurata, T. Sercu, K. Audhkhasi, S. Thomas, D. Dimitriadis, X. Cui, B. Ramabhadran, M. Picheny, L.-L. Lim, B. Roomi, and P. Hall, “English conversational telephone speech recognition by humans and machines,” in INTERSPEECH , 2017
2017
Earlier work this paper cites.
W. Xiong, L. Wu, J. Zhang, and A. Stolcke, “Session-level language modeling for conversational speech,” in EMNLP , 2018
2018
Earlier work this paper cites.
A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, and I. Sutskever, “Language models are unsupervised multitask learners,” OpenAI, Tech. Rep., 2019
2019
Earlier work this paper cites.
J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “BERT: Pre-training of deep bidirectional transformers for language understanding,” in NAACL , 2019
2019
Earlier work this paper cites.
2019
Earlier work this paper cites.
I. Tenney, D. Das, and E. Pavlick, “BERT rediscovers the classical NLP pipeline,” in ACL , 2019
2019
Cited alongside, same era.
I. Tenney, P. Xia, B. Chen, A. Wang, A. Poliak, R. T. McCoy, N. Kim, B. V. Durme, S. R. Bowman, D. Das, and E. Pavlick, “What do you learn from context? probing for sentence structure in contextualized word representations,” in ICLR , 2019
2019
Cited alongside, same era.
F. Petroni, T. Rocktäschel, S. Riedel, P. Lewis, A. Bakhtin, Y. Wu, and A. Miller, “Language models as knowledge bases?” in EMNLP-IJCNLP , 2019
2019
Cited alongside, same era.
J. Shin, Y. Lee, and K. Jung, “Effective sentence scoring method using bert for speech recognition,” in ACML , 2019
2019
Cited alongside, same era.
K. Irie, A. Zeyer, R. Schlüter, and H. Ney, “Training language models for long-span cross-sentence evaluation,” ASRU , 2019
2019
T. Wolf, L. Debut, V. Sanh, J. Chaumond, C. Delangue, A. Moi, P. Cistac, T. Rault, R. Louf, M. Funtowicz, J. Davison, S. Shleifer, P. von Platen, C. Ma, Y. Jernite, J. Plu, C. Xu, T. Le Scao, S. Gugger, M. Drame, Q. Lhoest, and A. Rush, “Transformers: State-of-the-art natural language processing,” in EMNLP: System Demonstrations , 2020
2020
Later among the works it cites.
G. Saon, Z. Tüske, and K. Audhkhasi, “Alignment-length synchronous decoding for RNN transducer,” in ICASSP , 2020
2020
Later among the works it cites.
A. Warstadt, A. Parrish, H. Liu, A. Mohananey, W. Peng, S.-F. Wang, and S. R. Bowman, “BLiMP: The benchmark of linguistic minimal pairs for English,” TACL , 2020
2020
Later among the works it cites.
H. Futami, H. Inaguma, S. Ueno, M. Mimura, S. Sakai, and T. Kawahara, “Distilling the knowledge of BERT for sequence-to-sequence ASR,” in INTERSPEECH , 2020
2020
Later among the works it cites.
alphaXiv searches the wider corpus for related work and actual follow-ups.
alphaXiv is searching for related work…
Cited alongside, same era.
A. Wang and K. Cho, “BERT has a mouth, and it must speak: BERT as a Markov random field language model,” in NeuralGen , 2019
2019
Cited alongside, same era.
D. S. Park, W. Chan, Y. Zhang, C.-C. Chiu, B. Zoph, E. D. Cubuk, and Q. V. Le, “SpecAugment: A simple data augmentation method for automatic speech recognition,” in INTERSPEECH , 2019
2019
Cited alongside, same era.
G. Saon, Z. Tüske, K. Audhkhasi, and B. Kingsbury, “Sequence noise injected training for end-to-end speech recognition,” in ICASSP , 2019
2019
Cited alongside, same era.
J. Salazar, D. Liang, T. Q. Nguyen, and K. Kirchhoff, “Masked language model scoring,” in ACL , 2020
2020
Cited alongside, same era.
K. Li, Z. Liu, T. He, H. Huang, F. Peng, D. Povey, and S. Khudanpur, “An empirical study of transformer-based neural language model adaptation,” ICASSP , 2020
2020
Cited alongside, same era.
A. Gulati, C.-C. Chiu, J. Qin, J. Yu, N. Parmar, R. Pang, S. Wang, W. Han, Y. Wu, Y. Zhang, and Z. Zhang, “Conformer: Convolution-augmented transformer for speech recognition,” in INTERSPEECH , 2020
2020
Cited alongside, same era.
Z. Tüske, G. Saon, K. Audhkhasi, and B. Kingsbury, “Single headed attention based sequence-to-sequence model for state-of-the-art results on Switchboard-300,” in INTERSPEECH , 2020
2020
Cited alongside, same era.
S. Ruder, “Recent Advances in Language Model Fine-tuning,” http://ruder.io/recent-advances-lm-fine-tuning, 2021
2021
Later among the works it cites.
S.-H. Chiu and B. Chen, “Innovative BERT-based reranking language models for speech recognition,” in IEEE SLT , 2021
2021
Later among the works it cites.
X. Zheng, C. Zhang, and P. C. Woodland, “Adapting GPT, GPT-2 and BERT language models for speech recognition,” ASRU , 2021
2021
Later among the works it cites.
H. Futami, H. Inaguma, M. Mimura, S. Sakai, and T. Kawahara, “ASR rescoring and confidence estimation with electra,” in ASRU , 2021
2021
Later among the works it cites.
2021
Later among the works it cites.
L. Xu, Y. Gu, J. Kolehmainen, H. Khan, A. Gandhe, A. Rastrow, A. Stolcke, and I. Bulyko, “Rescorebert: Discriminative speech recognition rescoring with bert,” in ICASSP , 2022
2022
Closest in time.
Y. Kubo, S. Karita, and M. Bacchiani, “Knowledge transfer from large-scale pretrained language models to end-to-end speech recognizers,” in ICASSP , 2022
2022
Closest in time.