2019

Effectiveness of self-supervised pre-training for speech recognition

Baevski, Alexei, Auli, Michael, Mohamed, Abdelrahman

Understand

We compare self-supervised representation learning algorithms which either explicitly quantize the audio data or learn representations without quantization.

We find the former to be more accurate since it builds a good vocabulary of the data through vq-wav2vec [1] to enable learning of effective representations in subsequent BERT training.
Different to previous work, we directly fine-tune the pre-trained BERT models on transcribed speech using a Connectionist Temporal Classification (CTC) loss instead of feeding the representations into a task-specific model.
We also propose a BERT-style model learning directly from the continuous audio data and compare pre-training on raw audio to spectral features.

Built on

Parallel Distributed Processing: Explorations in the Microstructure of Cognition, Vol. 1
G. E. Hinton, J. L. McClelland, and D. E. Rumelhart, · 1986
Earlier work this paper cites.
“Unsupervised pattern discovery in speech,”
A. Park and J. Glass, · 2008
Earlier work this paper cites.
“Towards spoken term discovery at scale with zero resources.,”
Aren Jansen, Kenneth Church, and Hynek Hermansky, · 2010
Earlier work this paper cites.
“The kaldi speech recognition toolkit,”
Daniel Povey, Arnab Ghoshal, Gilles Boulianne, Lukas Burget, Ondrej Glembek, Nagendra Goel, Mirko Hannemann, Petr Motlicek, Yanmin Qian, Petr Schwarz, Jan Silovsky, Georg Stemmer, and Karel Vesely, · 2011
Earlier work this paper cites.
“Towards unsupervised speech processing,”
J. Glass, · 2012
Earlier work this paper cites.
“Representation learning: A review and new perspectives,”
Yoshua Bengio, Aaron Courville, and Pascal Vincent, · 2013
Earlier work this paper cites.
“A summary of the 2012 JHU CLSP workshop on zero resource speech technologies and models of early language acquisition,”
Aren Jansen, Emmanuel Dupoux, Sharon Goldwater, Mark Johnson, Sanjeev Khudanpur, Kenneth Church, Naomi Feldman, Hynek Hermansky, Florian Metze, Richard Rose, et al., · 2013
Earlier work this paper cites.
“Semi-supervised training of deep neural networks,”
K. Vesely, M. Hannemann, and L. Burget, · 2013
Earlier work this paper cites.
“Distributed representations of words and phrases and their compositionality,”
Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean, · 2013
Earlier work this paper cites.
“Fixed-dimensional acoustic embeddings of variable-length segments in low-resource settings,”
Keith Levin, Katharine Henry, Aren Jansen, and K. Livescu, · 2013
Earlier work this paper cites.
“Multilingual acoustic models using distributed deep neural networks,”
Georg Heigold, Vincent Vanhoucke, Alan Senior, Patrick Nguyen, Marc’Aurelio Ranzato, Matthieu Devin, and Jeffrey Dean, · 2013
Earlier work this paper cites.
“Multilingual training of deep neural networks,”
A. Ghoshal, P. Swietojanski, and S. Renals, · 2013
Earlier work this paper cites.
“Cross-language knowledge transfer using multilingual deep neural network with shared hidden layers,”
Jui-Ting Huang, Jinyu Li, Dong Yu, Li Deng, and Yifan Gong, · 2013
Earlier work this paper cites.
“Word embeddings for speech recognition,”
Samy Bengio and Georg Heigold, · 2014
Earlier work this paper cites.
“Multilingual deep neural network based acoustic modeling for rapid language adaptation,”
Ngoc Thang Vu et. al., · 2014
Earlier work this paper cites.
“Deep learning,”
Yann LeCun, Yoshua Bengio, and Geoffrey Hinton, · 2015
Earlier work this paper cites.
“Librispeech: An asr corpus based on public domain audio books,”
Vassil Panayotov, Guoguo Chen, Daniel Povey, and Sanjeev Khudanpur, · 2015
Earlier work this paper cites.

Similar

“Unsupervised neural network based feature extraction using weak top-down constraints,”
Herman Kamper, Micha Elsner, Aren Jansen, and Sharon Goldwater, · 2015
Cited alongside, same era.
“Multilingual representations for low resource speech recognition and keyword search,”
Jia Cui, Brian Kingsbury, Bhuvana Ramabhadran, Abhinav Sethy, Kartik Audhkhasi, Xiaodong Cui, Ellen Kislal, Lidia Mangu, Markus Nussbaum-Thom, Michael Picheny, et al., · 2015
Cited alongside, same era.
“Audio word2vec: Unsupervised learning of audio segment representations using sequence-to-sequence autoencoder,”
Yu-An Chung, Chao-Chung Wu, Chia-Hao Shen, Hung yi Lee, and L. Lee, · 2016
Cited alongside, same era.
“Multi-view recurrent neural acoustic word embeddings,”
Wanjia He, Weiran Wang, and Karen Livescu, · 2016
Cited alongside, same era.
“Semi-supervised and cross-lingual knowledge transfer learnings for dnn hybrid acoustic models under low-resource conditions,”
Haihua Xu, Hang Su, Chongjia Ni, Xiong Xiao, Hao Huang, Eng Siong Chng, and Haizhou Li, · 2016
Cited alongside, same era.
“Semi-supervised ensemble dnn acoustic model training,”
Sheng Li, Xugang Lu, Shinsuke Sakai, Masato Mimura, and Tatsuya Kawahara, · 2017
Cited alongside, same era.
“Representations of language in a model of visually grounded speech signal,”
Grzegorz Chrupała, Lieke Gelderloos, and Afra Alishahi, · 2017
Cited alongside, same era.
“Visually grounded learning of keyword prediction from untranscribed speech,”
Herman Kamper, Shane Settle, Gregory Shakhnarovich, and Karen Livescu, · 2017
Cited alongside, same era.
“Neural discrete representation learning,”
Original
Aaron van den Oord, Oriol Vinyals, and Koray Kavukcuoglu, · 2017
Cited alongside, same era.
“Linguistic unit discovery from multi-modal inputs in unwritten languages: Summary of the ”speaking rosetta” JSALT 2017 workshop,”
Odette Scharenborg, Laurent Besacier, Alan Black, Mark Hasegawa-Johnson, Florian Metze, Graham Neubig, Sebastian Stüker, Pierre Godard, Markus Müller, Lucas Ondel, et al., · 2018
Cited alongside, same era.
“Speech2vec: A sequence-to-sequence framework for learning word embeddings from speech,”
Yu-An Chung and James Glass, · 2018
Cited alongside, same era.
“Representation learning with contrastive predictive coding,”
Aäron van den Oord, Yazhe Li, and Oriol Vinyals, · 2018
Cited alongside, same era.

Then

“Libri-light: A benchmark for asr with limited or no supervision,”
Original
Jacob Kahn et al., · 2019
Closest in time.
“Rwth asr systems for librispeech: Hybrid vs attention,”
Christoph Lüscher, Eugen Beck, Kazuki Irie, Markus Kitza, Wilfried Michel, Albert Zeyer, Ralf Schlüter, and Hermann Ney, · 2019
Closest in time.
“Roberta: A robustly optimized bert pretraining approach,”
Original
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov, · 2019
Closest in time.
“Transformers with convolutional context for asr,”
Abdelrahman Mohamed, Dmytro Okhonko, and Luke Zettlemoyer, · 2019
Closest in time.
“Learning representations by maximizing mutual information across views,”
Original
Philip Bachman, R Devon Hjelm, and William Buchwalter, · 2019
Closest in time.
“Specaugment: A simple data augmentation method for automatic speech recognition,”
Daniel S. Park, William Chan, Yu Zhang, Chung-Cheng Chiu, Barret Zoph, Ekin D. Cubuk, and Quoc V. Le, · 2019
Closest in time.
“fairseq: A fast, extensible toolkit for sequence modeling,”
Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli, · 2019
Closest in time.
“Transformer-xl: Attentive language models beyond a fixed-length context,”
Original
Zihang Dai, Zhilin Yang, Yiming Yang, Jaime Carbonell, Quoc V. Le, and Ruslan Salakhutdinov, · 2019
Closest in time.
“Wav2letter++: A fast open-source speech recognition system,”
V. Pratap, A. Hannun, Q. Xu, J. Cai, J. Kahn, G. Synnaeve, V. Liptchinsky, and R. Collobert, · 2019
Closest in time.
“Transformer-based acoustic modeling for hybrid speech recognition,”
Original
Yongqiang Wang, Abdelrahman Mohamed, Duc Le, Chunxi Liu, Alex Xiao, Jay Mahadeokar, Hongzhao Huang, Andros Tjandra, Xiaohui Zhang, Frank Zhang, Christian Fuegen, Geoffrey Zweig, and Michael L. Seltzer, · 2019
Closest in time.
“On the choice of modeling unit for sequence-to-sequence speech recognition,”
Kazuki Irie, Rohit Prabhavalkar, Anjuli Kannan, Antoine Bruguier, David Rybach, and Patrick Nguyen, · 2019
Closest in time.
“Self-training for end-to-end speech recognition,”
Original
Jacob Kahn, Ann Lee, and Awni Hannun, · 2019
Closest in time.
“Unsupervised learning of efficient and robust speech representations,” 2019
Kazuya Kawakami, Luyu Wang, Chris Dyer, Phil Blunsom, and Aaron van den Oord, · 2019
Closest in time.
“Acoustically grounded word embeddings for improved acoustics-to-word speech recognition,”
Shane Settle, Kartik Audhkhasi, Karen Livescu, and Michael Picheny, · 2019
Closest in time.
“An unsupervised autoregressive model for speech representation learning,”
Yu-An Chung, Wei-Ning Hsu, Hao Tang, and James R. Glass, · 2019
Closest in time.
“vq-wav2vec: Self-supervised learning of discrete speech representations,”
Alexei Baevski, Steffen Schneider, and Michael Auli, · 2020
Closest in time.
“wav2vec: Unsupervised pre-training for speech recognition,”
Steffen Schneider, Alexei Baevski, Ronan Collobert, and Michael Auli, · 2020
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…