2020

Improved Noisy Student Training for Automatic Speech Recognition

Park, Daniel S., Zhang, Yu, Jia, Ye et al.

Understand

Recently, a semi-supervised learning method known as "noisy student training" has been shown to improve image classification performance of deep networks significantly.

Noisy student training is an iterative self-training method that leverages augmentation to improve network performance.
In this work, we adapt and improve noisy student training for automatic speech recognition, employing (adaptive) SpecAugment as the augmentation method.
We find effective methods to filter, balance and augment the data generated in between self-training iterations.

Built on

H. Scudder, “Probability of error of some adaptive pattern-recognition machines,”
1965
Earlier work this paper cites.
D. Yarowsky, “Unsupervised word sense disambiguation rivaling supervised methods,” in
1995
Earlier work this paper cites.
G. Zavaliagkos and T. Colthurst, “Utilizing untranscribed training data to improve performance,” in
1998
Earlier work this paper cites.
L. Lamel, J. luc Gauvain, and G. Adda, “Lightly supervised acoustic model training,” in
2000
Earlier work this paper cites.
E. Riloff and J. Wiebe, “Learning extraction patterns for subjective expressions,” in
2003
Earlier work this paper cites.
S. Novotney and R. Schwartz, “Analysis of low-resource acoustic model self-training,” in
2009
Earlier work this paper cites.
A. Graves, “Sequence transduction with recurrent neural networks,” in
2012
Earlier work this paper cites.
M. Schuster and K. Nakajima, “Japanese and korean voice search,” in
2012
Earlier work this paper cites.
S. Thomas, M. L. Seltzer, K. Church, and H. Hermansky, “Deep neural network features and semi-supervised training for low resource speech recognition,” in
2013
Earlier work this paper cites.
Y. Shinohara, “A submodular optimization approach to sentence set selection,” in
2014
Earlier work this paper cites.
Ç. Gülçehre, O. Firat, K. Xu, K. Cho, L. Barrault, H. Lin, F. Bougares, H. Schwenk, and Y. Bengio, “On using monolingual corpora in neural machine translation,” in
2015
Earlier work this paper cites.
V. Panayotov, G. Chen, D. Povey, and S. Khudanpur, “Librispeech: An ASR corpus based on public domain audio books,” in
2015
Earlier work this paper cites.
H. Sak, A. Senior, K. Rao, and F. Beaufays, “Fast and accurate recurrent neural network acoustic models for speech recognition,” in
2015
Earlier work this paper cites.

Similar

W. Chan, N. Jaitly, Q. V. Le, and O. Vinyals, “Listen, Attend and Spell: A Neural Network for Large Vocabulary Conversational Speech Recognition,” in
2016
Cited alongside, same era.
J. Chorowski and N. Jaitly, “Towards better decoding and language model integration in sequence to sequence models,” in
2017
Cited alongside, same era.
W.-N. Hsu and J. Glass, “Extracting domain invariant features by unsupervised learning for robust automatic speech recognition,” in
2018
Cited alongside, same era.
Y.-A. Chung and J. Glass, “Speech2vec: A sequenceto-sequence framework for learning word embeddings from speech,” in
2018
Cited alongside, same era.
D. S. Park, W. Chan, Y. Zhang, C.-C. Chiu, B. Zoph, E. D. Cubuk, and Q. V. Le, “Specaugment: A simple data augmentation method for automatic speech recognition,” in
2019
Cited alongside, same era.
D. S. Park, Y. Zhang, C.-C. Chiu, Y. Chen, B. Li, W. Chan, Q. V. Le, and Y. Wu, “Specaugment on large scale datasets,” in
2019
Cited alongside, same era.
J. Kahn, M. Rivière, W. Zheng, E. Kharitonov, Q. Xu, P. Mazaré, J. Karadayi, V. Liptchinsky, R. Collobert, C. Fuegen, T. Likhomanenko, G. Synnaeve, A. Joulin, A. Mohamed, and E. Dupoux, “Libri-light: A benchmark for asr with limited or no supervision,” in
2019
Cited alongside, same era.
B. Li, T. N. Sainath, R. Pang, and Z. Wu, “Semi-supervised training for end-to-end models via weak distillation,” in
2019
Cited alongside, same era.
G. Synnaeve, Q. Xu, J. Kahn, T. Likhomanenko, E. Grave, V. Pratap, A. Sriram, V. Liptchinsky, and R. Collobert, “End-to-end asr: from supervised to semi-supervised learning with modern architectures,” in
2019
Cited alongside, same era.
S. H. K. Parthasarathi and N. Strom, “Lessons from building acoustic models with a million hours of speech,” in
2019
Cited alongside, same era.
Q. Xie, Z. Dai, E. Hovy, M.-T. Luong, and Q. V. Le, “Unsupervised data augmentation for consistency training,” in
2019
Cited alongside, same era.
X. Zhai, A. Oliver, A. Kolesnikov, and L. Beyer, “S4l: Self-supervised semi-supervised learning,” in
2019
Cited alongside, same era.

Then

J. Chorowski, R. J. Weiss, S. Bengio, and A. van den Oord, “Unsupervised speech representation learning using wavenet autoencoders,” in
2019
Later among the works it cites.
S. Schneider, A. Baevski, R. Collober, and M. Auli, “wav2vec: Unsupervised pre-training for speech recognition,” in
2019
Later among the works it cites.
A. Baevski, S. Schneider, and M. Auli, “vq-wav2vec: Self-supervised learning of discrete speech representations,” in
2019
Later among the works it cites.
S. Ling, Y. Liu, J. Salazar, and K. Kirchhoff, “Deep contextualized acoustic representations for semi-supervised speech recognition,” in
2019
Later among the works it cites.
K. Irie, R. Prabhavalkar, A. Kannan, A. Bruguier, D. Rybach, and P. Nguyen, “Model Unit Exploration for Sequence-to-Sequence Speech Recognition,” in
2019
Later among the works it cites.
C. Lüscher, E. Beck, K. Irie, M. Kitza, W. Michel, A. Zeyer, R. Schlüter, and H. Ney, “RWTH ASR systems for librispeech: Hybrid vs attention - w/o data augmentation,” in
2019
Later among the works it cites.
Q. Xie, M.-T. Luong, E. Hovy, and Q. V. Le, “Self-training with noisy student improves imagenet classification,” in
2020
Closest in time.
W. Han, Y. Zhang, J. Yu, Z. Zhang, J. Qin, A. Gulati, C. cheng Chiu, R. Pang, and Y. Wu, “ContextNet: Convolutional Neural Networks with Larger Context for Speech Recognition,” in
2020
Closest in time.
J. Kahn, A. Lee, and A. Hannun, “Self-training for end-to-end speech recognition,” in
2020
Closest in time.
W.-N. Hsu, A. Lee, G. Synnaeve, and A. Hannun, “Self-supervised speech recognition via local prior matching,” in
2020
Closest in time.
J. He, J. Gu, J. Shen, and M. Ranzato, “Revisiting self-training for neural sequence generation,” in
2020
Closest in time.
K. Sohn, D. Berthelot, C.-L. Li, Z. Zhang, N. Carlini, E. D. Cubuk, A. Kurakin, H. Zhang, and C. Raffel, “Fixmatch: Simplifying semi-supervised learning with consistency and confidence,” in
2020
Closest in time.
Q. Zhang, H. Lu, H. Sak, A. Tripathi, E. McDermott, S. Koo, and S. Kumar, “Transformer transducer: A streamable speech recognition model with transformer encoders and rnn-t loss,” in
2020
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…