2020

Multi-task self-supervised learning for Robust Speech Recognition

Ravanelli, Mirco, Zhong, Jianyuan, Pascual, Santiago et al.

Understand

Despite the growing interest in unsupervised learning, extracting meaningful knowledge from unlabelled audio remains an open challenge.

To take a step in this direction, we recently proposed a problem-agnostic speech encoder (PASE), that combines a convolutional encoder followed by multiple neural networks, called workers, tasked to solve self-supervised problems (i.e., ones that do not require manual annotations as ground truth).
PASE was shown to capture relevant speech information, including speaker voice-print and phonemes.
This paper proposes PASE+, an improved version of PASE for robust speech recognition in noisy and reverberant environments.

Built on

“Image method for efficiently simulating small‐room acoustics,”
J. B. Allen and D. A. Berkley, · 1979
Earlier work this paper cites.
“Phoneme recognition using time-delay neural networks,”
A. Waibel, T. Hanazawa, G. Hinton, K. Shikano, and K. J. Lang, · 1989
Earlier work this paper cites.
“DARPA TIMIT Acoustic Phonetic Continuous Speech Corpus CDROM,” 1993
J. S. Garofolo, L. F. Lamel, W. M. Fisher, J. G. Fiscus, D. S. Pallett, and N. L. Dahlgren, · 1993
Earlier work this paper cites.
“Greedy layer-wise training of deep networks,”
Y. Bengio, P. Lamblin, D. Popovici, and H. Larochelle, · 2006
Earlier work this paper cites.
“A fast learning algorithm for deep belief nets,”
G.E. Hinton, S. Osindero, and Y. Teh, · 2006
Earlier work this paper cites.
“Gammatone features and feature combination for large vocabulary speech recognition,”
R. Schluter, I. Bezrukov, H. Wagner, and H. Ney, · 2007
Earlier work this paper cites.
“The Kaldi Speech Recognition Toolkit,”
D. Povey et al., · 2011
Earlier work this paper cites.
“Multi-level adaptive networks in tandem and hybrid ASR systems,”
P. Bell, P. Swietojanski, and S. Renals, · 2013
Earlier work this paper cites.
“Auto-encoding variational Bayes,”
D. P. Kingma and M. Welling, · 2014
Earlier work this paper cites.
“Generative adversarial nets,”
I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio, · 2014
Earlier work this paper cites.
“The DIRHA-ENGLISH corpus and related tasks for distant-speech recognition in domestic environments,”
M. Ravanelli, L. Cristoforetti, R. Gretter, M. Pellin, A. Sosi, and M. Omologo, · 2015
Earlier work this paper cites.
“Contaminated speech training methods for robust DNN-HMM distant speech recognition,”
M. Ravanelli and M. Omologo, · 2015
Earlier work this paper cites.
“Batch normalization: Accelerating deep network training by reducing internal covariate shift,”
S. Ioffe and C. Szegedy, · 2015
Earlier work this paper cites.

Similar

“Delving deep into rectifiers: Surpassing human-level performance on ImageNet classification,”
K. He, X. Zhang, S. Ren, and J. Sun, · 2015
Cited alongside, same era.
“Adam: A method for stochastic optimization,”
D.P. Kingma and J. Ba, · 2015
Cited alongside, same era.
“Librispeech: An ASR corpus based on public domain audio books,”
V. Panayotov, G. Chen, D. Povey, and S. Khudanpur, · 2015
Cited alongside, same era.
“A time delay neural network architecture for efficient modeling of long temporal contexts,”
V. Peddinti, D. Povey, and S. Khudanpur, · 2015
Cited alongside, same era.
“Shuffle and learn: Unsupervised learning using temporal order verification,”
I. Misra, C. L. Zitnick, and M. Hebert, · 2016
Cited alongside, same era.
“Colorful image colorization,”
R. Zhang, P. Isola, and A. A. Efros, · 2016
Cited alongside, same era.
“Unsupervised learning of visual representations by solving jigsaw puzzle,”
M. Noroozi and P. Favaro., · 2016
Cited alongside, same era.
“Realistic multi-microphone data simulation for distant speech recognition,”
M. Ravanelli, P. Svaizer, and M. Omologo, · 2016
Cited alongside, same era.
“Multi-task self-supervised visual learning,”
C. Doersch and A. Zisserman, · 2017
Cited alongside, same era.
“Quasi-Recurrent Neural Networks,”
J. Bradbury, S. Merity, C. Xiong, and R. Socher, · 2017
Cited alongside, same era.
“Improving speech recognition by revising gated recurrent units,”
M. Ravanelli, P. Brakel, M. Omologo, and Y. Bengio, · 2017
Cited alongside, same era.
“BERT: pre-training of deep bidirectional transformers for language understanding,”
Original
J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, · 2018
Cited alongside, same era.

Then

“Representation learning with contrastive predictive coding,”
A. van den Oord, Y. Li, and O. Vinyals, · 2018
Later among the works it cites.
“Interpretable convolutional filters with SincNet,”
M. Ravanelli and Y. Bengio, · 2018
Later among the works it cites.
“Mutual information neural estimation,”
M. I. Belghazi, A. Baratin, S. Rajeshwar, S. Ozair, Y. Bengio, A. Courville, and D. Hjelm, · 2018
Later among the works it cites.
“Learning deep representations by mutual information estimation and maximization,”
D. Hjelm, A. Fedorov, S. Lavoie-Marchildon, K. Grewal, A. Trischler, and Y. Bengio, · 2018
Later among the works it cites.
“The fifth ‘CHiME’ speech separation and recognition challenge: Dataset, task and baselines,”
E. Vincent J. Barker, S. Watanabe and J. Trmal, · 2018
Later among the works it cites.
“Unsupervised speech representation learning using WaveNet autoencoders,”
Original
J. Chorowski, R. J. Weiss, S. Bengio, and A. van den Oord, · 2019
Later among the works it cites.
“Learning speaker representations with mutual information,”
M. Ravanelli and Y. Bengio, · 2019
Later among the works it cites.
“Learning problem-agnostic speech representations from multiple self-supervised tasks,”
S. Pascual, M. Ravanelli, J. Serrà, A. Bonafonte, and Y. Bengio, · 2019
Later among the works it cites.
“SpecAugment: A simple augmentation method for automatic speech recognition,”
B. Zoph, C.-C. Chiu, D. S. Park, E. D. Cubuk, Q. V. Le, W. Chan, and Y. Zhang, · 2019
Later among the works it cites.
“Multi-objective training of generative adversarial networks with multiple discriminators,”
I. Albuquerque, J. Monteiro, T. Doan, B. Considine, T. Falk, and I. Mitliagkas, · 2019
Later among the works it cites.
“Rethinking learning rate schedules for stochastic optimization,”
R. Ge, S. M. Kakade, R. Kidambi, and P. Netrapalli, · 2019
Later among the works it cites.
“The PyTorch-Kaldi Speech Recognition Toolkit,”
M. Ravanelli, T. Parcollet, and Y. Bengio, · 2019
Later among the works it cites.
“ALBERT: A lite BERT for self-supervised learning of language representations,”
Z. Lan, M. Chen, S. Goodman, K. Gimpel, P. Sharma, and R. Soricut, · 2020
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…