2019

Unsupervised pre-training for sequence to sequence speech recognition

Fan, Zhiyun, Zhou, Shiyu, Xu, Bo

Understand

This paper proposes a novel approach to pre-train encoder-decoder sequence-to-sequence (seq2seq) model with unpaired speech and transcripts respectively.

Our pre-training method is divided into two stages, named acoustic pre-trianing and linguistic pre-training.
In the acoustic pre-training stage, we use a large amount of speech to pre-train the encoder by predicting masked speech feature chunks with its context.
In the linguistic pre-training stage, we generate synthesized speech from a large number of transcripts using a single-speaker text to speech (TTS) system, and use the synthesized paired data to pre-train decoder.

Built on

“Language-independent and language-adaptive acoustic modeling for speech recognition,”
Tanja Schultz and Alex Waibel, · 2001
Earlier work this paper cites.
“Connectionist temporal classification: labelling unsegmented sequence data with recurrent neural networks,”
Alex Graves, Santiago Fernández, Faustino Gomez, and Jürgen Schmidhuber, · 2006
Earlier work this paper cites.
“Hkust/mts: A very large scale mandarin telephone speech corpus,”
Yi Liu, Pascale Fung, Yongsheng Yang, Christopher Cieri, Shudong Huang, and David Graff, · 2006
Earlier work this paper cites.
“Deep belief networks for phone recognition,”
Abdel-rahman Mohamed, George Dahl, and Geoffrey Hinton, · 2009
Earlier work this paper cites.
“Cross-lingual speech recognition under runtime resource constraints,”
Dong Yu, Li Deng, Peng Liu, Jian Wu, Yifan Gong, and Alex Acero, · 2009
Earlier work this paper cites.
“Sequence transduction with recurrent neural networks,”
Original
Alex Graves, · 2012
Earlier work this paper cites.
“Towards end-to-end speech recognition with recurrent neural networks,”
Alex Graves and Navdeep Jaitly, · 2014
Earlier work this paper cites.
“Learning phrase representations using rnn encoder-decoder for statistical machine translation,”
Original
Kyunghyun Cho, Bart Van Merriënboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio, · 2014
Earlier work this paper cites.
“Neural machine translation by jointly learning to align and translate,”
Original
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio, · 2014
Earlier work this paper cites.

Similar

“Attention-based models for speech recognition,”
Jan K Chorowski, Dzmitry Bahdanau, Dmitriy Serdyuk, Kyunghyun Cho, and Yoshua Bengio, · 2015
Cited alongside, same era.
“On using monolingual corpora in neural machine translation,”
Original
Caglar Gulcehre, Orhan Firat, Kelvin Xu, Kyunghyun Cho, Loic Barrault, Huei-Chi Lin, Fethi Bougares, Holger Schwenk, and Yoshua Bengio, · 2015
Cited alongside, same era.
“End-to-end attention-based large vocabulary speech recognition,”
Dzmitry Bahdanau, Jan Chorowski, Dmitriy Serdyuk, Philemon Brakel, and Yoshua Bengio, · 2016
Cited alongside, same era.
“Towards better decoding and language model integration in sequence to sequence models,”
Original
Jan Chorowski and Navdeep Jaitly, · 2016
Cited alongside, same era.
“Rethinking the inception architecture for computer vision,”
Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jon Shlens, and Zbigniew Wojna, · 2016
Cited alongside, same era.
“Attention is all you need,”
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin, · 2017
Cited alongside, same era.
“Aishell-1: An open-source mandarin speech corpus and a speech recognition baseline,”
Hui Bu, Jiayu Du, Xingyu Na, Bengu Wu, and Hao Zheng, · 2017
Cited alongside, same era.
“Multilingual recurrent neural networks with residual learning for low-resource speech recognition,”
Shiyu Zhou, Yuanyuan Zhao, Shuang Xu, Bo Xu, et al., · 2017
Cited alongside, same era.

Then

“Unsupervised cross-modal alignment of speech and text embedding spaces,”
Yu-An Chung, Wei-Hung Weng, Schrasing Tong, and James Glass, · 2018
Later among the works it cites.
“Semi-supervised end-to-end speech recognition.,”
Shigeki Karita, Shinji Watanabe, Tomoharu Iwata, Atsunori Ogawa, and Marc Delcroix, · 2018
Later among the works it cites.
“Bert: Pre-training of deep bidirectional transformers for language understanding,”
Original
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova, · 2018
Later among the works it cites.
“Aishell-2: Transforming mandarin asr research into industrial scale,”
Original
Jiayu Du, Xingyu Na, Xuechen Liu, and Hui Bu, · 2018
Later among the works it cites.
“Completely unsupervised phoneme recognition by a generative adversarial network harmonized with iteratively refined hidden markov models,”
Original
Kuan-Yu Chen, Che-Ping Tsai, Da-Rong Liu, Hung-Yi Lee, and Lin-shan Lee, · 2019
Closest in time.
“Semi-supervised training for end-to-end models via weak distillation,”
Bo Li, Tara N Sainath, Ruoming Pang, and Zelin Wu, · 2019
Closest in time.
“What does bert learn about the structure of language?,”
Ganesh Jawahar, Benoît Sagot, Djamé Seddah, Samuel Unicomb, Gerardo Iñiguez, Márton Karsai, Yannick Léo, Márton Karsai, Carlos Sarraute, Éric Fleury, et al., · 2019
Closest in time.
“Boosting character-based chinese speech synthesis via multi-task learning and dictionary tutoring,”
Yuxiang Zou, Linhao Dong, and Bo Xu, · 2019
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…