2021

Citrinet: Closing the Gap between Non-Autoregressive and Autoregressive End-to-End Models for Automatic Speech Recognition

Majumdar, Somshubra, Balam, Jagadeesh, Hrinchuk, Oleksii et al.

Understand

We propose Citrinet - a new end-to-end convolutional Connectionist Temporal Classification (CTC) based automatic speech recognition (ASR) model.

Citrinet is deep residual neural model which uses 1D time-channel separable convolutions combined with sub-word encoding and squeeze-and-excitation.
The resulting architecture significantly reduces the gap between non-autoregressive and sequence-to-sequence and transducer models.
We evaluate Citrinet on LibriSpeech, TED-LIUM2, AISHELL-1 and Multilingual LibriSpeech (MLS) English speech datasets.

Built on

A. Graves, S. Fernández, F. Gomez, and J. Schmidhuber, “Connectionist temporal classification: labelling unsegmented sequence data with recurrent neural networks,” in ICML , 2006
2006
Earlier work this paper cites.
A. Graves, “Sequence transduction with recurrent neural networks,” arXiv:1211.3711 , 2012
Original
2012
Earlier work this paper cites.
J. Chorowski, D. Bahdanau, K. Cho, and Y. Bengio, “End-to-end continuous speech recognition using attention-based Recurrent NN: First results,” in NIPS, Deep Learning and Representation Learning Workshop , 2014
2014
Earlier work this paper cites.
A. Rousseau, P. Deléglise, and Y. Estève, “Enhancing the TED-LIUM corpus with selected data for language modeling and more TED talks,” in Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC’14) , 2014
2014
Earlier work this paper cites.
V. Panayotov, G. Chen, D. Povey, and S. Khudanpur, “Librispeech: an ASR corpus based on public domain audio books,” in ICASSP , 2015, pp. 5206–5210
2015
Earlier work this paper cites.
W. Chan, N. Jaitly, Q. V. Le, and O. Vinyals, “Listen, attend and spell,” in ICASSP , 2016
2016
Earlier work this paper cites.
D. Bahdanau, J. Chorowski, D. Serdyuk, P. Brakel, and Y. Bengio, “End-to-end attention-based large vocabulary speech recognition.” in ICASSP , 2016
2016
Earlier work this paper cites.
D. Amodei and etc, “Deep speech 2: End-to-end speech recognition in english and mandarin,” in ICML , 2016
2016
Earlier work this paper cites.
E. Battenberg, J. Chen, R. Child, A. Coates, Y. G. Y. Li, H. Liu, S. Satheesh, A. Sriram, and Z. Zhu, “Exploring neural transducers for end-to-end speech recognition,” in ASRU , 2017
2017
Earlier work this paper cites.
R. Prabhavalkar, K. Rao, T. N. Sainath, B. Li, L. Johnson, and N. Jaitly, “A comparison of sequence-to-sequence models for speech recognition,” in Interspeech , 2017
2017
Earlier work this paper cites.
B. Hui, D. Jiayu, N. Xingyu, W. Bengu, and H. Zheng, “AIShell-1: An open-source mandarin speech corpus and a speech recognition baseline,” in Oriental COCOSDA , 2017
2017
Earlier work this paper cites.

Similar

Baidu Research blog, “Deep Speech 3: Even more end-to-end speech recognition,” http://research.baidu.com/Blog/index-view?id=90 , 2017
2017
Cited alongside, same era.
J. Hu, L. Shen, and G. Sun, “Squeeze-and-excitation networks,” in ICVPR , 2018
2018
Cited alongside, same era.
T. Kudo and J. Richardson, “SentencePiece: A simple and language independent subword tokenizer and detokenizer for neural text processing,” arXiv:1808.06226 , 2018
Original
2018
Cited alongside, same era.
K. Han, A. Chandrashekaran, J. Kim, and I. R. Lane, “The CAPIO 2017 conversational speech recognition system,” arXiv:1801.00059 , 2018
Original
2018
Cited alongside, same era.
G. Synnaeve, Q. Xu, J. Kahn, E. Grave, T. Likhomanenko, V. Pratap, A. Sriram, V. Liptchinsky, and R. Collobert, “End-to-end ASR: from supervised to semi-supervised learning with modern architectures,” arXiv:1911.08460 , 2019
Original
2019
Cited alongside, same era.
D. Park, W. Chan, Y. Zhang, C. Chiu, B. Zoph, E. Cubuk, and Q. Le, “SpecAugment: A simple data augmentation method for automatic speech recognition,” arXiv:1904.08779 , 2019
Original
2019
Cited alongside, same era.
B. Ginsburg, P. Castonguay, O. Hrinchuk, O. Kuchaiev, V. Lavrukhin, R. Leary, J. Li, H. Nguyen, and C. J. M., “Stochastic gradient methods with layer-wise adaptive moments for training of deep networks,” arXiv:1905.11286 , 2019
Original
2019
Cited alongside, same era.
S. Karita, N. Chen, T. Hayashi, T. Hori, H. Inaguma, Z. Jiang, M. Someki, N. E. Y. Soplin, R. Yamamoto, X. Wang, S. Watanabe, T. Yoshimura, and W. Zhang, “A comparative study on Transformer vs RNN in speech applications,” in ASRU Workshop , 2019
2019
Cited alongside, same era.
O. Kuchaiev, J. Li, H. Nguyen, O. Hrinchuk, R. Leary, B. Ginsburg, S. Kriman, S. Beliaev, V. Lavrukhin, J. Cook et al. , “NeMo: a toolkit for building AI applications using neural modules,” arXiv:1909.09577 , 2019
Original
2019
Cited alongside, same era.
S. Kriman, S. Beliaev, B. Ginsburg, J. Huang, O. Kuchaiev, V. Lavrukhin, R. Leary, J. Li, and Y. Zhang, “QuartzNet: Deep automatic speech recognition with 1d time-channel separable convolutions,” in ICASSP , 2020
2020
Cited alongside, same era.

Then

T. Likhomanenko, Q. Xu, V. Pratap, P. Tomasello, J. Kahn, G. Avidov, R. Collobert, and G. Synnaeve, “Rethinking evaluation in ASR: Are our models robust enough?” arXiv:2010.11745 , 2020
Original
2020
Later among the works it cites.
P. Guo, F. Boyer, X. Chang, T. Hayashi, Y. Higuchi, H. Inaguma, N. Kamo, C. Li, D. Garcia-Romero, J. Shi et al. , “Recent developments on ESPnet toolkit boosted by Conformer,” arXiv:2010.13956 , 2020
Original
2020
Later among the works it cites.
Q. Zhang, H. Lu, H. Sak, A. Tripathi, E. McDermott, S. Koo, and S. Kumar, “Transformer Transducer: A streamable speech recognition model with Transformer encoders and RNN-T loss,” in ICASSP , 2020
2020
Later among the works it cites.
W. Han, Z. Zhang, Y. Zhang, J. Yu, C.-C. Chiu, J. Qin, A. Gulati, R. Pang, and Y. Wu, “Contextnet: Improving convolutional neural networks for automatic speech recognition with global context,” arXiv:2005.03191 , 2020
Original
2020
Later among the works it cites.
A. Gulati, J. Qin, C.-C. Chiu, N. Parmar, Y. Zhang, J. Yu, W. Han, S. Wang, Z. Zhang, Y. Wu, and R. Pang, “Conformer: Convolution-augmented transformer for speech recognition,” arXiv:2005.08100 , 2020
Original
2020
Later among the works it cites.
D. Jurafsky and J. H. Martin, Speech and Language Processing . Preprint, 2020
2020
Later among the works it cites.
V. Pratap, Q. Xu, A. Sriram, G. Synnaeve, and R. Collobert, “MLS: A large-scale multilingual dataset for speech research,” Interspeech , 2020
2020
Later among the works it cites.
T. Wolf, L. Debut, V. Sanh, J. Chaumond, C. Delangue, A. Moi, P. Cistac, T. Rault, R. Louf, M. Funtowicz, J. Davison, S. Shleifer, P. von Platen, C. Ma, Y. Jernite, J. Plu, C. Xu, T. L. Scao, S. Gugger, M. Drame, Q. Lhoest, and A. M. Rush, “HuggingFace’s Transformers: State-of-the-art natural language processing,” arXiv:1910.03771 , 2020
Original
2020
Later among the works it cites.
W. Zhou, W. Michel, K. Irie, M. Kitza, R. Schlüter, and H. Ney, “The RWTH ASR System for TED-LIUM Release 2: Improving hybrid HMM with SpecAugment,” in ICASSP , 2020
2020
Later among the works it cites.
B. Zhang, D. Wu, Z. Yao, X. Wang, F. Yu, C. Yang, L. Guo, Y. Hu, L. Xie, and X. Lei, “Unified streaming and non-streaming two-pass end-to-end model for speech recognition,” arXiv:2012.05481 , 2020
Original
2020
Later among the works it cites.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…