2020

SlimIPL: Language-Model-Free Iterative Pseudo-Labeling

Likhomanenko, Tatiana, Xu, Qiantong, Kahn, Jacob et al.

Understand

Recent results in end-to-end automatic speech recognition have demonstrated the efficacy of pseudo-labeling for semi-supervised models trained both with Connectionist Temporal Classification (CTC) and Sequence-to-Sequence (seq2seq) losses.

Iterative Pseudo-Labeling (IPL), which continuously trains a single model using pseudo-labels iteratively re-generated as the model learns, has been shown to further improve performance in ASR.
We improve upon the IPL algorithm: as the model learns, we propose to iteratively re-generate transcriptions with hard labels (the most probable tokens), that is, without a language model.
We call this approach Language-Model-Free IPL (slimIPL) and give a resultant training setup for low-resource settings with CTC-based models.

Built on

H. Scudder, “Probability of error of some adaptive pattern-recognition machines,” IEEE Transactions on Information Theory , vol. 11, no. 3, pp. 363–371, 1965
1965
Earlier work this paper cites.
J. Godfrey and E. Holliman, “Switchboard-1 release 2 LDC97S62,” Philadelphia: LDC , 1993
1993
Earlier work this paper cites.
D. Yarowsky, “Unsupervised word sense disambiguation rivaling supervised methods,” in 33rd annual meeting of the association for computational linguistics , 1995, pp. 189–196
1995
Earlier work this paper cites.
LDC et al. , “2000 hub5 english evaluation speech LDC2002S09 and transcripts LDC2002T43,” Web Download. Philadelphia: LDC , 2002
2002
Earlier work this paper cites.
C. Cieri, , D. Graff, O. Kimball, D. Miller, and K. Walker, “Fisher english training speech parts 1 and 2 transcripts LDC200{4,5}T19,” Philadelphia: LDC , 2004, 2005
2005
Earlier work this paper cites.
C. Cieri, D. Miller, and K. Walker, “Fisher english training speech parts 1 and 2 LDC200{4,5}S13,” Philadelphia: LDC , 2004, 2005
2005
Earlier work this paper cites.
A. Graves, S. Fernández, F. Gomez, and J. Schmidhuber, “Connectionist temporal classification: labelling unsegmented sequence data with recurrent neural networks,” in Proceedings of the 23rd international conference on Machine learning , 2006, pp. 369–376
2006
Earlier work this paper cites.
D. McClosky, E. Charniak, and M. Johnson, “Effective self-training for parsing,” in Proceedings of the Human Language Technology Conference of the NAACL, Main Conference , 2006, pp. 152–159
2006
Earlier work this paper cites.
N. Ueffing, “Using monolingual source-language data to improve mt performance,” in International Workshop on Spoken Language Translation (IWSLT) 2006 , 2006
2006
Earlier work this paper cites.
R. Reichart and A. Rappoport, “Self-training for enhancement and domain adaptation of statistical parsers trained on small datasets,” in Proceedings of the 45th Annual Meeting of the Association of Computational Linguistics , 2007, pp. 616–623
2007
Earlier work this paper cites.
J. G. Fiscus et al. , “2003 nist rich transcription evaluation data LDC2007S10,” Web Download. Philadelphia: LDC , 2007
2007
Earlier work this paper cites.
Z. Huang and M. Harper, “Self-training pcfg grammars with latent annotations across languages,” in Proceedings of the 2009 conference on empirical methods in natural language processing , 2009, pp. 832–841
2009
Earlier work this paper cites.
S. Novotney and R. Schwartz, “Analysis of low-resource acoustic model self-training,” in Tenth Annual Conference of the International Speech Communication Association , 2009
2009
Earlier work this paper cites.
O. Chapelle, B. Schölkopf, and A. Zien, Semi-supervised Learning . Mit Press, 2010
2010
Earlier work this paper cites.
J. Duchi, E. Hazan, and Y. Singer, “Adaptive subgradient methods for online learning and stochastic optimization,” Journal of machine learning research , vol. 12, no. Jul, pp. 2121–2159, 2011
2011
Earlier work this paper cites.
D. Povey, A. Ghoshal, G. Boulianne, L. Burget, O. Glembek, N. Goel, M. Hannemann, P. Motlicek, Y. Qian, P. Schwarz et al. , “The kaldi speech recognition toolkit,” in IEEE 2011 workshop on automatic speech recognition and understanding , no. CONF. IEEE Signal Processing Society, 2011
2011
Earlier work this paper cites.
D.-H. Lee, “Pseudo-label: The simple and efficient semi-supervised learning method for deep neural networks,” in Workshop on challenges in representation learning, ICML , vol. 3, no. 2, 2013
2013
Earlier work this paper cites.
V. Panayotov, G. Chen, D. Povey, and S. Khudanpur, “Librispeech: an asr corpus based on public domain audio books,” in 2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) . IEEE, 2015, pp. 5206–5210
2015
Earlier work this paper cites.

Similar

J. Zhang and C. Zong, “Exploiting source-side monolingual data in neural machine translation,” in Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing , 2016, pp. 1535–1545
2016
Cited alongside, same era.
R. Sennrich, B. Haddow, and A. Birch, “Improving neural machine translation models with monolingual data,” in Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , 2016, pp. 86–96
2016
Cited alongside, same era.
R. Collobert, C. Puhrsch, and G. Synnaeve, “Wav2letter: an end-to-end convnet-based speech recognition system,” arXiv preprint arXiv:1609.03193 , 2016
Original
2016
Cited alongside, same era.
A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin, “Attention is all you need,” in Advances in neural information processing systems , 2017, pp. 5998–6008
2017
Cited alongside, same era.
A. v. d. Oord, Y. Li, and O. Vinyals, “Representation learning with contrastive predictive coding,” arXiv preprint arXiv:1807.03748 , 2018
Original
2018
Cited alongside, same era.
K. Imamura, A. Fujita, and E. Sumita, “Enhancement of encoder and attention using target monolingual corpora in neural machine translation,” in Proceedings of the 2nd Workshop on Neural Machine Translation and Generation , 2018, pp. 55–63
2018
Cited alongside, same era.
S. Edunov, M. Ott, M. Auli, and D. Grangier, “Understanding back-translation at scale,” in Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing , 2018, pp. 489–500
2018
Cited alongside, same era.
I. Z. Yalniz, H. Jégou, K. Chen, M. Paluri, and D. Mahajan, “Billion-scale semi-supervised learning for image classification,” arXiv preprint arXiv:1905.00546 , 2019
Original
2019
Cited alongside, same era.
S. H. K. Parthasarathi and N. Strom, “Lessons from building acoustic models with a million hours of speech,” in ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) . IEEE, 2019, pp. 6670–6674
2019
Cited alongside, same era.
D. S. Park, W. Chan, Y. Zhang, C.-C. Chiu, B. Zoph, E. D. Cubuk, and Q. V. Le, “Specaugment: A simple data augmentation method for automatic speech recognition,” Proc. Interspeech 2019 , pp. 2613–2617, 2019
2019
Cited alongside, same era.
D. Berthelot, N. Carlini, I. Goodfellow, N. Papernot, A. Oliver, and C. A. Raffel, “Mixmatch: A holistic approach to semi-supervised learning,” in Advances in Neural Information Processing Systems , vol. 32, 2019
2019
Cited alongside, same era.
D. Berthelot, N. Carlini, E. D. Cubuk, A. Kurakin, K. Sohn, H. Zhang, and C. Raffel, “Remixmatch: Semi-supervised learning with distribution alignment and augmentation anchoring,” arXiv preprint arXiv:1911.09785 , 2019
Original
2019
Cited alongside, same era.

Then

G. Synnaeve, Q. Xu, J. Kahn, T. Likhomanenko, E. Grave, V. Pratap, A. Sriram, V. Liptchinsky, and R. Collobert, “End-to-end asr: from supervised to semi-supervised learning with modern architectures,” in Workshop on Self-supervision in Audio and Speech, ICML , 2020
2020
Closest in time.
D. S. Park, Y. Zhang, Y. Jia, W. Han, C.-C. Chiu, B. Li, Y. Wu, and Q. V. Le, “Improved noisy student training for automatic speech recognition,” Proc. Interspeech 2020 , pp. 2817–2821, 2020
2020
Closest in time.
Y. Zhang, J. Qin, D. S. Park, W. Han, C.-C. Chiu, R. Pang, Q. V. Le, and Y. Wu, “Pushing the limits of semi-supervised learning for automatic speech recognition,” arXiv preprint arXiv:2010.10504 , 2020
Original
2020
Closest in time.
Q. Xie, M.-T. Luong, E. Hovy, and Q. V. Le, “Self-training with noisy student improves imagenet classification,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2020, pp. 10 687–10 698
2020
Closest in time.
J. He, J. Gu, J. Shen, and M. Ranzato, “Revisiting self-training for neural sequence generation,” in International Conference on Learning Representations , 2020
2020
Closest in time.
J. Pino, Q. Xu, X. Ma, M. J. Dousti, and Y. Tang, “Self-training for end-to-end speech translation,” Proc. Interspeech 2020 , pp. 1476–1480, 2020
2020
Closest in time.
E. Arazo, D. Ortego, P. Albert, N. E. O’Connor, and K. McGuinness, “Pseudo-labeling and confirmation bias in deep semi-supervised learning,” in 2020 International Joint Conference on Neural Networks (IJCNN) . IEEE, 2020, pp. 1–8
2020
Closest in time.
Y. Chen, W. Wang, and C. Wang, “Semi-supervised asr by end-to-end self-training,” Proc. Interspeech 2020 , pp. 2787–2791, 2020
2020
Closest in time.
A. Baevski, Y. Zhou, A. Mohamed, and M. Auli, “wav2vec 2.0: A framework for self-supervised learning of speech representations,” Advances in Neural Information Processing Systems , vol. 33, 2020
2020
Closest in time.
K. Sohn, D. Berthelot, N. Carlini, Z. Zhang, H. Zhang, C. A. Raffel, E. D. Cubuk, A. Kurakin, and C.-L. Li, “Fixmatch: Simplifying semi-supervised learning with consistency and confidence,” Advances in Neural Information Processing Systems , vol. 33, 2020
2020
Closest in time.
S. Ling, Y. Liu, J. Salazar, and K. Kirchhoff, “Deep contextualized acoustic representations for semi-supervised speech recognition,” in ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) . IEEE, 2020, pp. 6429–6433
2020
Closest in time.
A. Fan, E. Grave, and A. Joulin, “Reducing transformer depth on demand with structured dropout,” in International Conference on Learning Representations , 2020
2020
Closest in time.
T. Likhomanenko, Q. Xu, V. Pratap, P. Tomasello, J. Kahn, G. Avidov, R. Collobert, and G. Synnaeve, “Rethinking evaluation in asr: Are our models robust enough?” arXiv preprint arXiv:2010.11745 , 2020
Original
2020
Closest in time.
Q. Xu, A. Baevski, T. Likhomanenko, P. Tomasello, A. Conneau, R. Collobert, G. Synnaeve, and M. Auli, “Self-training and pre-training are complementary for speech recognition,” in ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) . IEEE, 2021, pp. 3030–3034
2021
Closest in time.
C. Talnikar, T. Likhomanenko, R. Collobert, and G. Synnaeve, “Joint masked cpc and ctc training for asr,” in ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) . IEEE, 2021, pp. 3045–3049
2021
Closest in time.
H. Touvron, M. Cord, M. Douze, F. Massa, A. Sablayrolles, and H. Jégou, “Training data-efficient image transformers & distillation through attention,” in International Conference on Machine Learning . PMLR, 2021, pp. 10 347–10 357
2021
Closest in time.
V. Manohar, T. Likhomanenko, Q. Xu, W.-N. Hsu, R. Collobert, Y. Saraf, G. Zweig, and A. Mohamed, “Kaizen: Continuously improving teacher using exponential moving average for semi-supervised speech recognition,” arXiv preprint arXiv:2106.07759 , 2021
Original
2021
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…