2020

Joint Masked CPC and CTC Training for ASR

Talnikar, Chaitanya, Likhomanenko, Tatiana, Collobert, Ronan et al.

Understand

Self-supervised learning (SSL) has shown promise in learning representations of audio that are useful for automatic speech recognition (ASR).

But, training SSL models like wav2vec~2.0 requires a two-stage pipeline.
In this paper we demonstrate a single-stage training of ASR models that can utilize both unlabeled and labeled data.
During training, we alternately minimize two losses: an unsupervised masked Contrastive Predictive Coding (CPC) loss and the supervised audio-to-text alignment loss Connectionist Temporal Classification (CTC).

Built on

“Connectionist temporal classification: labelling unsegmented sequence data with recurrent neural networks,”
Alex Graves, Santiago Fernández, Faustino Gomez, and Jürgen Schmidhuber, · 2006
Earlier work this paper cites.
“Deep neural network features and semi-supervised training for low resource speech recognition,”
Samuel Thomas, Michael L Seltzer, Kenneth Church, and Hynek Hermansky, · 2013
Earlier work this paper cites.
“Adam: A method for stochastic optimization,”
Original
Diederik P Kingma and Jimmy Ba, · 2014
Earlier work this paper cites.
“Librispeech: an asr corpus based on public domain audio books,”
Vassil Panayotov, Guoguo Chen, Daniel Povey, and Sanjeev Khudanpur, · 2015
Earlier work this paper cites.
“Speech enhancement and recognition using multi-task learning of long short-term memory recurrent neural networks,”
Zhuo Chen, Shinji Watanabe, Hakan Erdogan, and John R Hershey, · 2015
Earlier work this paper cites.
“Deep speech 2: End-to-end speech recognition in english and mandarin,”
Dario Amodei, Sundaram Ananthanarayanan, Rishita Anubhai, Jingliang Bai, Eric Battenberg, Carl Case, Jared Casper, Bryan Catanzaro, Qiang Cheng, Guoliang Chen, et al., · 2016
Earlier work this paper cites.
“Adversarial multi-task learning of deep neural networks for robust speech recognition.,”
Yusuke Shinohara, · 2016
Earlier work this paper cites.
“Attention is all you need,”
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin, · 2017
Earlier work this paper cites.
“Joint ctc-attention based end-to-end speech recognition using multi-task learning,”
Suyoun Kim, Takaaki Hori, and Shinji Watanabe, · 2017
Earlier work this paper cites.

Similar

“Representation learning with contrastive predictive coding,”
Original
Aaron van den Oord, Yazhe Li, and Oriol Vinyals, · 2018
Cited alongside, same era.
“Bert: Pre-training of deep bidirectional transformers for language understanding,”
Original
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova, · 2018
Cited alongside, same era.
“wav2letter++: The fastest open-source speech recognition system,”
Original
Vineel Pratap, Awni Hannun, Qiantong Xu, Jeff Cai, Jacob Kahn, Gabriel Synnaeve, Vitaliy Liptchinsky, and Ronan Collobert, · 2018
Cited alongside, same era.
“Improved asr for under-resourced languages through multi-task learning with acoustic landmarks,”
Original
Di He, Boon Pang Lim, Xuesong Yang, Mark Hasegawa-Johnson, and Deming Chen, · 2018
Cited alongside, same era.
“Specaugment: A simple data augmentation method for automatic speech recognition,”
Original
Daniel S Park, William Chan, Yu Zhang, Chung-Cheng Chiu, Barret Zoph, Ekin D Cubuk, and Quoc V Le, · 2019
Cited alongside, same era.
“Who needs words? lexicon-free speech recognition,”
Tatiana Likhomanenko, Gabriel Synnaeve, and Ronan Collobert, · 2019
Cited alongside, same era.
“End-to-end asr: from supervised to semi-supervised learning with modern architectures,”
Original
Gabriel Synnaeve, Qiantong Xu, Jacob Kahn, Tatiana Likhomanenko, Edouard Grave, Vineel Pratap, Anuroop Sriram, Vitaliy Liptchinsky, and Ronan Collobert, · 2019
Cited alongside, same era.
“End-to-end asr: from supervised to semi-supervised learning with modern architectures,” 2020
Gabriel Synnaeve, Qiantong Xu, Jacob Kahn, Tatiana Likhomanenko, Edouard Grave, Vineel Pratap, Anuroop Sriram, Vitaliy Liptchinsky, and Ronan Collobert, · 2020
Cited alongside, same era.
“Improved noisy student training for automatic speech recognition,” 2020
Daniel S. Park, Yu Zhang, Ye Jia, Wei Han, Chung-Cheng Chiu, Bo Li, Yonghui Wu, and Quoc V. Le, · 2020
Cited alongside, same era.

Then

“Data augmenting contrastive learning of speech representations in the time domain,”
Original
Eugene Kharitonov, Morgane Rivière, Gabriel Synnaeve, Lior Wolf, Pierre-Emmanuel Mazaré, Matthijs Douze, and Emmanuel Dupoux, · 2020
Closest in time.
“Libri-light: A benchmark for asr with limited or no supervision,”
Jacob Kahn, Morgane Rivière, Weiyi Zheng, Evgeny Kharitonov, Qiantong Xu, Pierre-Emmanuel Mazaré, Julien Karadayi, Vitaliy Liptchinsky, Ronan Collobert, Christian Fuegen, et al., · 2020
Closest in time.
“wav2vec 2.0: A framework for self-supervised learning of speech representations,”
Original
Alexei Baevski, Henry Zhou, Abdelrahman Mohamed, and Michael Auli, · 2020
Closest in time.
“Reducing transformer depth on demand with structured dropout,”
Angela Fan, Edouard Grave, and Armand Joulin, · 2020
Closest in time.
“A simple framework for contrastive learning of visual representations,”
Original
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton, · 2020
Closest in time.
“Self-supervised learning of pretext-invariant representations,”
Ishan Misra and Laurens van der Maaten, · 2020
Closest in time.
“Semi-supervised speech recognition via local prior matching,”
Original
Wei-Ning Hsu, Ann Lee, Gabriel Synnaeve, and Awni Hannun, · 2020
Closest in time.
“Iterative pseudo-labeling for speech recognition,”
Original
Qiantong Xu, Tatiana Likhomanenko, Jacob Kahn, Awni Hannun, Gabriel Synnaeve, and Ronan Collobert, · 2020
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…