2020

fairseq S2T: Fast Speech-to-Text Modeling with fairseq

Wang, Changhan, Tang, Yun, Ma, Xutai et al.

Understand

We introduce fairseq S2T, a fairseq extension for speech-to-text (S2T) modeling tasks such as end-to-end speech recognition and speech-to-text translation.

It follows fairseq's careful design for scalability and extensibility.
We provide end-to-end workflows from data pre-processing, model training to offline (online) inference.
We implement state-of-the-art RNN-based, Transformer-based as well as Conformer-based models and open-source detailed training recipes.

Built on

Lingvo: a modular and scalable framework for sequence-to-sequence modeling
Original
Jonathan Shen, Patrick Nguyen, Yonghui Wu, Zhifeng Chen, Mia X Chen, Ye Jia, Anjuli Kannan, Tara Sainath, Yuan Cao, Chung-Cheng Chiu, et al. 2019 · 1902
Earlier work this paper cites.
Transformers with convolutional context for asr
Original
Abdelrahman Mohamed, Dmytro Okhonko, and Luke Zettlemoyer. 2019 · 1904
Earlier work this paper cites.
Thinking slow about latency evaluation for simultaneous machine translation
Original
Colin Cherry and George Foster. 2019 · 1906
Earlier work this paper cites.
Massively multilingual neural machine translation in the wild: Findings and challenges
Original
Naveen Arivazhagan, Ankur Bapna, Orhan Firat, Dmitry Lepikhin, Melvin Johnson, Maxim Krikun, Mia Xu Chen, Yuan Cao, George Foster, Colin Cherry, et al. 2019a · 1907
Earlier work this paper cites.
Roberta: A robustly optimized bert pretraining approach
Original
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019 · 1907
Earlier work this paper cites.
End-to-end asr: from supervised to semi-supervised learning with modern architectures
Original
Gabriel Synnaeve, Qiantong Xu, Jacob Kahn, Edouard Grave, Tatiana Likhomanenko, Vineel Pratap, Anuroop Sriram, Vitaliy Liptchinsky, and Ronan Collobert. 2019 · 1911
Earlier work this paper cites.
Bleu: a method for automatic evaluation of machine translation
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002 · 2002
Earlier work this paper cites.
Connectionist temporal classification: labelling unsegmented sequence data with recurrent neural networks
Alex Graves, Santiago Fernández, Faustino Gomez, and Jürgen Schmidhuber. 2006 · 2006
Earlier work this paper cites.
Self-training for end-to-end speech translation
Original
Juan Pino, Qiantong Xu, Xutai Ma, Mohammad Javad Dousti, and Yun Tang. 2020 · 2006
Earlier work this paper cites.
Improving cross-lingual transfer learning for end-to-end speech recognition with speech translation
Original
Changhan Wang, Juan Pino, and Jiatao Gu. 2020b · 2006
Earlier work this paper cites.
Self-supervised representations improve end-to-end speech translation
Original
Anne Wu, Changhan Wang, Juan Pino, and Jiatao Gu. 2020 · 2006
Earlier work this paper cites.
Simuleval: An evaluation toolkit for simultaneous translation
Original
Xutai Ma, Mohammad Javad Dousti, Changhan Wang, Jiatao Gu, and Juan Pino. 2020a · 2007
Earlier work this paper cites.
Covost 2 and massively multilingual speech-to-text translation
Changhan Wang, Anne Wu, and Juan Pino. 2020c · 2007
Earlier work this paper cites.
Pushing the limits of semi-supervised learning for automatic speech recognition
Original
Yu Zhang, James Qin, Daniel S Park, Wei Han, Chung-Cheng Chiu, Ruoming Pang, Quoc V Le, and Yonghui Wu. 2020 · 2010
Earlier work this paper cites.
The kaldi speech recognition toolkit
Daniel Povey, Arnab Ghoshal, Gilles Boulianne, Lukas Burget, Ondrej Glembek, Nagendra Goel, Mirko Hannemann, Petr Motlicek, Yanmin Qian, Petr Schwarz, et al. 2011 · 2011
Earlier work this paper cites.
Librispeech: an asr corpus based on public domain audio books
Vassil Panayotov, Guoguo Chen, Daniel Povey, and Sanjeev Khudanpur. 2015 · 2015
Earlier work this paper cites.
chrf: character n-gram f-score for automatic mt evaluation
Maja Popović. 2015 · 2015
Earlier work this paper cites.
Listen and translate: A proof of concept for end-to-end speech-to-text translation
Original
Alexandre Bérard, Olivier Pietquin, Christophe Servan, and Laurent Besacier. 2016 · 2016
Earlier work this paper cites.
Listen, attend and spell: A neural network for large vocabulary conversational speech recognition
William Chan, Navdeep Jaitly, Quoc Le, and Oriol Vinyals. 2016 · 2016
Earlier work this paper cites.

Similar

An attentional model for speech translation without transcription
Long Duong, Antonios Anastasopoulos, David Chiang, Steven Bird, and Trevor Cohn. 2016 · 2016
Cited alongside, same era.
Must-c: a multilingual speech translation corpus
Mattia A Di Gangi, Roldano Cattoni, Luisa Bentivogli, Matteo Negri, and Marco Turchi. 2019a · 2017
Cited alongside, same era.
OpenNMT: Open-source toolkit for neural machine translation
Guillaume Klein, Yoon Kim, Yuntian Deng, Jean Senellart, and Alexander Rush. 2017 · 2017
Cited alongside, same era.
A study on data augmentation of reverberant speech for robust speech recognition
Tom Ko, Vijayaditya Peddinti, Daniel Povey, Michael L Seltzer, and Sanjeev Khudanpur. 2017 · 2017
Cited alongside, same era.
Online and linear-time attention by enforcing monotonic alignments
Colin Raffel, Minh-Thang Luong, Peter J Liu, Ron J Weiss, and Douglas Eck. 2017 · 2017
Cited alongside, same era.
Attention is all you need
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017 · 2017
Cited alongside, same era.
Tied multitask learning for neural speech translation
Antonios Anastasopoulos and David Chiang. 2018 · 2018
Cited alongside, same era.
End-to-end automatic speech translation of audiobooks
Alexandre Bérard, Laurent Besacier, Ali Can Kocabiyikoglu, and Olivier Pietquin. 2018 · 2018
Cited alongside, same era.
Pykaldi: A python wrapper for kaldi
Dogan Can, Victor R. Martinez, Pavlos Papadopoulos, and Shrikanth S. Narayanan. 2018 · 2018
Cited alongside, same era.
Openseq2seq: extensible toolkit for distributed and mixed precision training of sequence-to-sequence models
Oleksii Kuchaiev, Boris Ginsburg, Igor Gitman, Vitaly Lavrukhin, Carl Case, and Paulius Micikevicius. 2018 · 2018
Cited alongside, same era.
SentencePiece: A simple and language independent subword tokenizer and detokenizer for neural text processing
Taku Kudo and John Richardson. 2018 · 2018
Cited alongside, same era.
Mixed precision training
Paulius Micikevicius, Sharan Narang, Jonah Alben, Gregory Diamos, Erich Elsen, David Garcia, Boris Ginsburg, Michael Houston, Oleksii Kuchaiev, Ganesh Venkatesh, et al. 2018 · 2018
Cited alongside, same era.

Then

Bytes are all you need: End-to-end multilingual speech recognition and synthesis with bytes
Bo Li, Yu Zhang, Tara Sainath, Yonghui Wu, and William Chan. 2019 · 2019
Later among the works it cites.
STACL: Simultaneous translation with implicit anticipation and controllable latency using prefix-to-prefix framework
Mingbo Ma, Liang Huang, Hao Xiong, Renjie Zheng, Kaibo Liu, Baigong Zheng, Chuanqiang Zhang, Zhongjun He, Hairong Liu, Xing Li, Hua Wu, and Haifeng Wang. 2019 · 2019
Later among the works it cites.
Facebook FAIR’s WMT19 news translation task submission
Nathan Ng, Kyra Yee, Alexei Baevski, Myle Ott, Michael Auli, and Sergey Edunov. 2019 · 2019
Later among the works it cites.
fairseq: A fast, extensible toolkit for sequence modeling
Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli. 2019 · 2019
Later among the works it cites.
Specaugment: A simple data augmentation method for automatic speech recognition
Daniel S. Park, William Chan, Yu Zhang, Chung-Cheng Chiu, Barret Zoph, Ekin D. Cubuk, and Quoc V. Le. 2019 · 2019
Later among the works it cites.
Pytorch: An imperative style, high-performance deep learning library
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. 2019 · 2019
Later among the works it cites.
wav2vec: Unsupervised Pre-Training for Speech Recognition
Steffen Schneider, Alexei Baevski, Ronan Collobert, and Michael Auli. 2019 · 2019
Later among the works it cites.
VizSeq: a visual analysis toolkit for text generation tasks
Changhan Wang, Anirudh Jain, Danlu Chen, and Jiatao Gu. 2019 · 2019
Later among the works it cites.
FINDINGS OF THE IWSLT 2020 EVALUATION CAMPAIGN
Ebrahim Ansari, Amittai Axelrod, Nguyen Bach, Ondřej Bojar, Roldano Cattoni, Fahim Dalvi, Nadir Durrani, Marcello Federico, Christian Federmann, Jiatao Gu, Fei Huang, Kevin Knight, Xutai Ma, Ajay Nagesh, Matteo Negri, Jan Niehues, Juan Pino, Elizabeth Salesky, Xing Shi, Sebastian Stüker, Marco Turchi, Alexander Waibel, and Changhan Wang. 2020 · 2020
Closest in time.
wav2vec 2.0: A framework for self-supervised learning of speech representations
Alexei Baevski, Yuhao Zhou, Abdelrahman Mohamed, and Michael Auli. 2020 · 2020
Closest in time.
Unsupervised cross-lingual representation learning at scale
Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020 · 2020
Closest in time.
Conformer: Convolution-augmented Transformer for Speech Recognition
Anmol Gulati, James Qin, Chung-Cheng Chiu, Niki Parmar, Yu Zhang, Jiahui Yu, Wei Han, Shibo Wang, Zhengdong Zhang, Yonghui Wu, and Ruoming Pang. 2020 · 2020
Closest in time.
Array programming with numpy
Charles R Harris, K Jarrod Millman, Stéfan J van der Walt, Ralf Gommers, Pauli Virtanen, David Cournapeau, Eric Wieser, Julian Taylor, Sebastian Berg, Nathaniel J Smith, et al. 2020 · 2020
Closest in time.
ESPnet-ST: All-in-one speech translation toolkit
Hirofumi Inaguma, Shun Kiyono, Kevin Duh, Shigeki Karita, Nelson Yalta, Tomoki Hayashi, and Shinji Watanabe. 2020 · 2020
Closest in time.
Self-training for end-to-end speech recognition
Jacob Kahn, Ann Lee, and Awni Hannun. 2020 · 2020
Closest in time.
BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020 · 2020
Closest in time.
Synchronous speech recognition and speech-to-text translation with interactive decoding
Yuchen Liu, Jiajun Zhang, Hao Xiong, Long Zhou, Zhongjun He, Hua Wu, Haifeng Wang, and Chengqing Zong. 2020 · 2020
Closest in time.
Monotonic multihead attention
Xutai Ma, Juan Pino, James Cross, Liezl Puzon, and Jiatao Gu. 2020b · 2020
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…