2020

Semi-Supervised Speech Recognition via Local Prior Matching

Hsu, Wei-Ning, Lee, Ann, Synnaeve, Gabriel et al.

Understand

For sequence transduction tasks like speech recognition, a strong structured prior model encodes rich information about the target space, implicitly ruling out invalid sequences by assigning them low probability.

In this work, we propose local prior matching (LPM), a semi-supervised objective that distills knowledge from a strong prior (e.g.
a language model) to provide learning signal to a discriminative model trained on unlabeled speech.
We demonstrate that LPM is theoretically well-motivated, simple to implement, and superior to existing knowledge distillation techniques under comparable settings.

Built on

Libri-light: A benchmark for asr with limited or no supervision
Original
Kahn, J., Rivière, M., Zheng, W., Kharitonov, E., Xu, Q., Mazaré, P.-E., Karadayi, J., Liptchinsky, V., Collobert, R., Fuegen, C., et al · 1912
Earlier work this paper cites.
Knowledge of language: Its nature, origin, and use
Chomsky, N · 1986
Earlier work this paper cites.
Towards phone segmentation for concatenative speech synthesis
Adell, J. and Bonafonte, A · 2004
Earlier work this paper cites.
Early language acquisition: cracking the speech code
Kuhl, P. K · 2004
Earlier work this paper cites.
Discriminative training for large vocabulary speech recognition
Povey, D · 2005
Earlier work this paper cites.
Posterior regularization for structured latent variable models
Ganchev, K., Gillenwater, J., Taskar, B., et al · 2010
Earlier work this paper cites.
Unsupervised speech segmentation: An analysis of the hypothesized phone boundaries
Scharenborg, O., Wan, V., and Ernestus, M · 2010
Earlier work this paper cites.
Towards unsupervised speech processing
Glass, J · 2012
Earlier work this paper cites.
Non-mainstream languages and speech recognition: Some challenges
Precoda, K · 2013
Earlier work this paper cites.
Sequence-discriminative training of deep neural networks
Veselỳ, K., Ghoshal, A., Burget, L., and Povey, D · 2013
Earlier work this paper cites.
Generative adversarial nets
Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., and Bengio, Y · 2014
Earlier work this paper cites.
Deep Speech: Scaling up end-to-end speech recognition
Original
Hannun, A., Case, C., Casper, J., Catanzaro, B., Diamos, G., Elsen, E., Prenger, R., Satheesh, S., Sengupta, S., Coates, A., et al · 2014
Earlier work this paper cites.
Dropout: a simple way to prevent neural networks from overfitting
Srivastava, N., Hinton, G., Krizhevsky, A., Sutskever, I., and Salakhutdinov, R · 2014
Earlier work this paper cites.
Librispeech: an ASR corpus based on public domain audio books
Panayotov, V., Chen, G., Povey, D., and Khudanpur, S · 2015
Earlier work this paper cites.
Deep Speech 2: End-to-end speech recognition in English and Mandarin
Amodei, D., Ananthanarayanan, S., Anubhai, R., Bai, J., Battenberg, E., Case, C., Casper, J., Catanzaro, B., Cheng, Q., Chen, G., et al · 2016
Earlier work this paper cites.
End-to-end attention-based large vocabulary speech recognition
Bahdanau, D., Chorowski, J., Serdyuk, D., Brakel, P., and Bengio, Y · 2016
Earlier work this paper cites.
Abstractive text summarization using sequence-to-sequence RNNs and beyond
Nallapati, R., Zhou, B., Gulcehre, C., Xiang, B., et al · 2016
Earlier work this paper cites.
Improving neural machine translation models with monolingual data
Sennrich, R., Haddow, B., and Birch, A · 2016
Earlier work this paper cites.

Similar

Towards better decoding and language model integration in sequence to sequence models
Chorowski, J. and Jaitly, N · 2017
Cited alongside, same era.
Knowledge distillation across ensembles of multilingual models for low-resource languages
Cui, J., Kingsbury, B., Ramabhadran, B., Saon, G., Sercu, T., Audhkhasi, K., Sethy, A., Nussbaum-Thom, M., and Rosenberg, A · 2017
Cited alongside, same era.
Language modeling with gated convolutional networks
Dauphin, Y. N., Fan, A., Auli, M., and Grangier, D · 2017
Cited alongside, same era.
Unsupervised domain adaptation for robust speech recognition via variational autoencoder-based data augmentation
Hsu, W.-N., Zhang, Y., and Glass, J · 2017
Cited alongside, same era.
Acoustic modeling for Google home
Li, B., Sainath, T. N., Narayanan, A., Caroselli, J., Bacchiani, M., Misra, A., Shafran, I., Sak, H., Pundak, G., Chin, K. K., et al · 2017
Cited alongside, same era.
Unsupervised adaptation with domain separation networks for robust speech recognition
Meng, Z., Chen, Z., Mazalov, V., Li, J., and Gong, Y · 2017
Cited alongside, same era.
An unsupervised deep domain adaptation approach for robust speech recognition
Sun, S., Zhang, B., Xie, L., and Zhang, Y · 2017
Cited alongside, same era.
Listening while speaking: Speech chain by deep learning
Tjandra, A., Sakti, S., and Nakamura, S · 2017
Cited alongside, same era.
Attention is all you need
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, Ł., and Polosukhin, I · 2017
Cited alongside, same era.
Semi-supervised DNN training with word selection for ASR
Veselỳ, K., Burget, L., and Cernockỳ, J · 2017
Cited alongside, same era.
Gate activation signal analysis for gated recurrent neural networks and its correlation with phoneme boundaries
Wang, Y.-H., Chung, C.-T., and Lee, H.-y · 2017
Cited alongside, same era.
Unpaired image-to-image translation using cycle-consistent adversarial networks
Zhu, J.-Y., Park, T., Isola, P., and Efros, A. A · 2017
Cited alongside, same era.

Then

Semi-supervised end-to-end speech recognition
Karita, S., Watanabe, S., Iwata, T., Ogawa, A., and Delcroix, M · 2018
Later among the works it cites.
Subword regularization: Improving neural network translation models with multiple subword candidates
Kudo, T · 2018
Later among the works it cites.
Sentencepiece: A simple and language independent subword tokenizer and detokenizer for neural text processing
Kudo, T. and Richardson, J · 2018
Later among the works it cites.
Semi-supervised training of acoustic models using lattice-free MMI
Manohar, V., Hadian, H., Povey, D., and Khudanpur, S · 2018
Later among the works it cites.
Adversarial teacher-student learning for unsupervised domain adaptation
Meng, Z., Li, J., Gong, Y., and Juang, B.-H · 2018
Later among the works it cites.
wav2letter++: The fastest open-source speech recognition system
Original
Pratap, V., Hannun, A., Xu, Q., Cai, J., Kahn, J., Synnaeve, G., Liptchinsky, V., and Collobert, R · 2018
Later among the works it cites.
Fully convolutional speech recognition
Original
Zeghidour, N., Xu, Q., Liptchinsky, V., Usunier, N., Synnaeve, G., and Collobert, R · 2018
Later among the works it cites.
Semi-supervised sequence-to-sequence ASR using unpaired speech and text
Baskar, M. K., Watanabe, S., Astudillo, R., Hori, T., Burget, L., and Černockỳ, J · 2019
Later among the works it cites.
A fully differentiable beam search decoder
Collobert, R., Hannun, A., and Synnaeve, G · 2019
Later among the works it cites.
Sequence-to-sequence speech recognition with time-depth separable convolutions
Hannun, A., Lee, A., Xu, Q., and Collobert, R · 2019
Later among the works it cites.
Cycle-consistency training for end-to-end speech recognition
Hori, T., Astudillo, R., Hayashi, T., Zhang, Y., Watanabe, S., and Le Roux, J · 2019
Later among the works it cites.
Semi-supervised training for end-to-end models via weak distillation
Li, B., Sainath, T. N., Pang, R., and Wu, Z · 2019
Later among the works it cites.
Adversarial training of end-to-end speech recognition using a criticizing language model
Liu, A. H., Lee, H.-y., and Lee, L.-s · 2019
Later among the works it cites.
Attentive adversarial learning for domain-invariant training
Meng, Z., Li, J., and Gong, Y · 2019
Later among the works it cites.
Lessons from building acoustic models with a million hours of speech
Parthasarathi, S. H. K. and Strom, N · 2019
Later among the works it cites.
Parallel neural text-to-speech
Original
Peng, K., Ping, W., Song, Z., and Zhao, K · 2019
Later among the works it cites.
End-to-end feedback loss in speech chain framework via straight-through estimator
Tjandra, A., Sakti, S., and Nakamura, S · 2019
Later among the works it cites.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…