2020

Unsupervised pretraining transfers well across languages

Rivière, Morgane, Joulin, Armand, Mazaré, Pierre-Emmanuel et al.

Understand

Cross-lingual and multi-lingual training of Automatic Speech Recognition (ASR) has been extensively investigated in the supervised setting.

This assumes the existence of a parallel corpus of speech and orthographic transcriptions.
Recently, contrastive predictive coding (CPC) algorithms have been proposed to pretrain ASR systems with unlabelled data.
In this work, we investigate whether unsupervised pretraining transfers well across languages.

Built on

“Language-independent and language-adaptive acoustic modeling for speech recognition,”
T. Schultz and A. Waibel, · 2001
Earlier work this paper cites.
“Learning a similarity metric discriminatively, with application to face verification,”
S. Chopra, R. Hadsell, and Y. LeCun, · 2005
Earlier work this paper cites.
“Cross-domain and cross-language portability of acoustic features estimated by multilayer perceptrons,”
A. Stolcke, F. Grezl, M.-Y. Hwang, X. Lei, N. Morgan, and D. Vergyri, · 2006
Earlier work this paper cites.
“Connectionist temporal classification: labelling unsegmented sequence data with recurrent neural networks,”
A. Graves, S. Fernández, F. Gomez, and J. Schmidhuber, · 2006
Earlier work this paper cites.
“Distance metric learning for large margin nearest neighbor classification,”
K. Q. Weinberger and L. K. Saul, · 2009
Earlier work this paper cites.
“Multilingual acoustic modeling for speech recognition based on subspace gaussian mixture models,”
L. Burget, P. Schwarz, M. Agarwal, P. Akyazi, K. Feng, A. Ghoshal, O. Glembek, N. Goel, M. Karafiát, and D. Povey, · 2010
Earlier work this paper cites.
“The language-independent bottleneck features,”
K. Veselỳ, M. Karafiát, F. Grézl, M. Janda, and E. Egorova, · 2012
Earlier work this paper cites.
“Multilingual acoustic models using distributed deep neural networks,”
G. Heigold, V. Vanhoucke, A. Senior, P. Nguyen, M. A. Ranzato, M. Devin, and J. Dean, · 2013
Earlier work this paper cites.
“Cross-language knowledge transfer using multilingual deep neural network with shared hidden layers,”
J.-T. Huang, J. Li, D. Yu, L. Deng, and Y. Gong, · 2013
Earlier work this paper cites.
“Distributed representations of words and phrases and their compositionality,”
T. Mikolov, I. Sutskever, K. Chen, G. Corrado, and J. Dean, · 2013
Earlier work this paper cites.

Similar

“Evaluating speech features with the minimal-pair abx task: Analysis of the classical mfc/plp pipeline,”
T. Schatz, V. Peddinti, F. Bach, A. Jansen, H. Hermansky, and E. Dupoux, · 2013
Cited alongside, same era.
“Discriminative unsupervised feature learning with convolutional neural networks,”
A. Dosovitskiy, J. T. Springenberg, M. Riedmiller, and T. Brox, · 2014
Cited alongside, same era.
“Facenet: A unified embedding for face recognition and clustering,”
F. Schroff, D. Kalenichenko, and J. Philbin, · 2015
Cited alongside, same era.
“Batch normalization: Accelerating deep network training by reducing internal covariate shift,”
Original
S. Ioffe and C. Szegedy, · 2015
Cited alongside, same era.
“Unsupervised feature extraction by time-contrastive learning and nonlinear ica,”
A. Hyvarinen and H. Morioka, · 2016
Cited alongside, same era.
“Unsupervised learning by predicting noise,”
P. Bojanowski and A. Joulin, · 2017
Cited alongside, same era.
“Attention is all you need,”
A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin, · 2017
Cited alongside, same era.
“The zero resource speech challenge 2017,”
Original
E. Dunbar, X. Cao, J. Benjumea, J. Karadayi, M. Bernard, L.t Besacier, X. Anguera, and E. Dupoux, · 2017
Cited alongside, same era.
“Feature optimized dpgmm clustering for unsupervised subword modeling: A contribution to zerospeech 2017,”
M. Heck, S. Sakti, and S. Nakamura, · 2017
Cited alongside, same era.
“Multilingually trained bottleneck features in spoken language recognition,”
R. Fer, P. Matějka, F. Grézl, O. Plchot, K. Veselỳ, and J. H. Černockỳ, · 2017
Cited alongside, same era.

Then

“Representation learning with contrastive predictive coding,”
Original
A. van den Oord, Y. Li, and O. Vinyals, · 2018
Later among the works it cites.
“Sequence-based multi-lingual low resource speech recognition,”
S. Dalmia, R. Sanabria, F. Metze, and A. W. Black, · 2018
Later among the works it cites.
“Learning deep representations by mutual information estimation and maximization,”
Original
R. D. Hjelm, A. Fedorov, S. Lavoie-Marchildon, K. Grewal, P. Bachman, A. Trischler, and Y. Bengio, · 2018
Later among the works it cites.
“wav2vec: Unsupervised pre-training for speech recognition,”
Original
S. Schneider, A. Baevski, R. Collobert, and M. Auli, · 2019
Later among the works it cites.
“Multilingual speech recognition with corpus relatedness sampling,”
Original
X. Li, S. Dalmia, A. W. Black, and F. Metze, · 2019
Later among the works it cites.
“Massively multilingual adversarial speech recognition,”
Original
O. Adams, M. Wiesner, S. Watanabe, and D. Yarowsky, · 2019
Later among the works it cites.
“Contrastive multiview coding,”
Original
Y. Tian, D. Krishnan, and P. Isola, · 2019
Later among the works it cites.
“Unsupervised speech representation learning using wavenet autoencoders,”
Original
J. Chorowski, R. J. Weiss, S. Bengio, and A. Oord, · 2019
Later among the works it cites.
“Libri-light: A benchmark for asr with limited or no supervision,”
J. Kahn, M. Rivière, W. Zheng, E. Kharitonov, Q. Xu, P.-E. Mazaré, J. Karadayi, V. Liptchinsky, R. Collobert, C. Fuegen, T. Likhomanenko, G. Synnaeve, A. Joulin, A. Mohamed, and E. Dupoux, · 2020
Closest in time.
“Learning robust and multilingual speech representations,” 2020
K. Kawakami, L. Wang, C. Dyer, P. Blunsom, and A. van den Oord, · 2020
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…