2017

Multilingual Speech Recognition With A Single End-To-End Model

Toshniwal, Shubham, Sainath, Tara N., Weiss, Ron J. et al.

Understand

Training a conventional automatic speech recognition (ASR) system to support multiple languages is challenging because the sub-word unit, lexicon and word inventories are typically language specific.

In contrast, sequence-to-sequence models are well suited for multilingual ASR because they encapsulate an acoustic, pronunciation and language model jointly in a single network.
In this work we present a single sequence-to-sequence ASR model trained on 9 different Indian languages, which have very little overlap in their scripts.
Specifically, we take a union of language-specific grapheme sets and train a grapheme-based sequence-to-sequence model jointly on data from all languages.

Built on

“A Study of Multilingual Speech Recognition,”
Fuliang Weng, Harry Bratt, Leonardo Neumeyer, and Andreas Stolcke, · 1997
Earlier work this paper cites.
“Fast Bootstrapping of LVCSR Systems with Multilingual Phoneme Sets,”
T. Schultz and A. Waibel, · 1997
Earlier work this paper cites.
“Long Short-Term Memory,”
Sepp Hochreiter and Jürgen Schmidhuber, · 1997
Earlier work this paper cites.
“Bidirectional Recurrent Neural Networks,”
Mike Schuster, Kuldip K. Paliwal, and A. General, · 1997
Earlier work this paper cites.
“Multitask Learning,”
Rich Caruana, · 1997
Earlier work this paper cites.
“Language-Independent and Language-Adaptive Acoustic Modeling for Speech Recognition,”
Tanja Schultz and Alex Waibel, · 2001
Earlier work this paper cites.
“A Simple Approach for Building Transliteration Editors for Indian Languages,”
Prahallad Lavanya, Prahallad Kishore, and Ganapa Thiraju Madhavi, · 2005
Earlier work this paper cites.
“Language-Dependent State Clustering for Multilingual Acoustic Modelling,”
Thomas Niesler, · 2007
Earlier work this paper cites.
“A Study on Multilingual Acoustic Modeling for Large Vocabulary ASR,”
Hui Lin, Li Deng, Dong Yu, Yi-fan Gong, Alex Acero, and Chin-Hui Lee, · 2009
Earlier work this paper cites.

Similar

“Multilingual Acoustic Modeling for Speech Recognition Based on Subspace Gaussian Mixture Models,”
Lukas Burget, Petr Schwarz, Mohit Agarwal, Pinar Akyazi, Kai Feng, Arnab Ghoshal, Ondrej Glembek, Nagendra Goel, Martin Karafiat, Daniel Povey, Ariya Rastrow, Richard C. Rose, and Samuel Thomas, · 2010
Cited alongside, same era.
“Practical Variational Inference for Neural Networks,”
Alex Graves, · 2011
Cited alongside, same era.
“Multilingual MLP Features for Low Resource LVCSR Systems,”
Samuel Thomas, Sriram Ganapathy, and Hynek Hermansky, · 2012
Cited alongside, same era.
“Large Scale Distributed Deep Networks,”
Jeffrey Dean, Greg S. Corrado, Rajat Monga, Kai Chen, Matthieu Devin, Quoc V. Le, Mark Z. Mao, Marc’Aurelio Ranzato, Andrew Senior, Paul Tucker, Ke Yang, and Andrew Y. Ng, · 2012
Cited alongside, same era.
“Multilingual Acoustic Models Using Distributed Deep Neural Networks,”
Georg Heigold, Vincent Vanhoucke, Andrew Senior, Patrick Nguyen, Marc’aurelio Ranzato, Matthieu Devin, and Jeff Dean, · 2013
Cited alongside, same era.
“Multilingual Training of Deep Neural Networks,”
Arnab Ghoshal, Pawel Swietojanski, and Steve Renals, · 2013
Cited alongside, same era.
“Cross-Language Knowledge Transfer Using Multilingual Deep Neural Network with Shared Hidden Layers,”
Jui-Ting Huang, Jinyu Li, Dong Yu, Li Deng, and Yifan Gong, · 2013
Cited alongside, same era.
“Investigation on Cross-and Multilingual MLP Features Under Matched and Mismatched Acoustical Conditions,”
Zoltán Tüske, Joel Pinto, Daniel Willett, and Ralf Schlüter, · 2013
Cited alongside, same era.
“Automatic Speech Recognition for Under-Resourced Languages: A Survey,”
Laurent Besacier, Etienne Barnard, Alexey Karpov, and Tanja Schultz, · 2014
Cited alongside, same era.

Then

“Multilingual Deep Neural Network Based Acoustic Modeling for Rapid Language Adaptation,”
N. T. Vu, D. Imseng, D. Povey, P. Motlicek, T. Schultz, and H. Bourlard, · 2014
Later among the works it cites.
“Multitask Learning of Deep Neural Networks for Low-Resource Speech Recognition,”
D. Chen and B. K. W. Mak, · 2015
Later among the works it cites.
“Fast and Accurate Recurrent Neural Network Acoustic Models for Speech Recognition,”
Hasim Sak, Andrew W. Senior, Kanishka Rao, and Françoise Beaufays, · 2015
Later among the works it cites.
“TensorFlow: Large-Scale Machine Learning on Heterogeneous Systems,” 2015
Martín Abadi et al, · 2015
Later among the works it cites.
“Listen, Attend and Spell: A Neural Network for Large Vocabulary Conversational Speech Recognition,”
William Chan, Navdeep Jaitly, Quoc V. Le, and Oriol Vinyals, · 2016
Later among the works it cites.
“Language Independent End-to-End Architecture for Joint Language and Speech Recognition,”
Shinji Watanabe, Takaaki Hori, and John Hershey, · 2017
Closest in time.
“An Investigation of Deep Neural Networks for Multilingual Speech Recognition Training and Adaptation,”
Sibo Tong, Philip N. Garner, and Hervé Bourlard, · 2017
Closest in time.
“A Comparison of Sequence-to-Sequence Models for Speech Recognition,”
Rohit Prabhavalkar, Kanishka Rao, Tara N. Sainath, Bo Li, Leif Johnson, and Navdeep Jaitly, · 2017
Closest in time.
“Towards Better Decoding and Language Model Integration in Sequence to Sequence Models,”
Jan Chorowski and Navdeep Jaitly, · 2017
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…