Fetching the paper…

Reading the bibliography…

2022

MAESTRO: Matched Speech Text Representations through Modality Matching

Chen, Zhehuai, Zhang, Yu, Rosenberg, Andrew et al.

Original
PDF
Post on X

Understand

We present Maestro, a self-supervised training method to unify representations learnt from speech and text modalities.

Self-supervised learning from speech signals aims to learn the latent structure inherent in the signal, while self-supervised learning from text attempts to capture lexical information.
Learning aligned representations from unpaired speech and text sequences is a challenging task.
Previous work either implicitly enforced the representations learnt from these two modalities to be aligned in the latent space through multitasking and parameter sharing or explicitly through conversion of modalities via speech synthesis.

MAESTRO: Matched Speech Text Representations through Modality Matching · Around

Built on

R. Kuhn and R. De Mori, “A cache-based natural language model for speech recognition,” IEEE transactions on pattern analysis and machine intelligence , vol. 12, no. 6, pp. 570–583, 1990
1990
Earlier work this paper cites.
A. Rousseau, P. Deléglise, and Y. Esteve, “TED-LIUM: an automatic speech recognition dedicated corpus.” in Proc. LREC , 2012, pp. 125–129
2012
Earlier work this paper cites.
M. J. Gales, K. M. Knill, A. Ragni, and S. P. Rath, “Speech recognition and keyword spotting for low-resource languages: Babel project research at cued,” in Fourth International workshop on spoken language technologies for under-resourced languages (SLTU-2014) , 2014, pp. 16–23
2014
Earlier work this paper cites.
V. Panayotov, G. Chen, D. Povey, and S. Khudanpur, “Librispeech: An ASR corpus based on public domain audio books,” in Proc. ICASSP , 2015
2015
Earlier work this paper cites.
A. Vaswani et al. , “Attention is all you need,” in Advances in neural information processing systems , 2017, pp. 5998–6008
2017
Earlier work this paper cites.
J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “BERT: Pre-training of deep bidirectional transformers for language understanding,” arXiv preprint arXiv:1810.04805 , 2018
Original
2018
Earlier work this paper cites.
A. Renduchintala, S. Ding, M. Wiesner, and S. Watanabe, “Multi-modal data augmentation for end-to-end ASR,” arXiv preprint arXiv:1803.10299 , 2018
Original
2018
Earlier work this paper cites.
A. Conneau, K. Khandelwal, N. Goyal, V. Chaudhary, G. Wenzek, F. Guzmán, E. Grave, M. Ott, L. Zettlemoyer, and V. Stoyanov, “Unsupervised cross-lingual representation learning at scale,” arXiv preprint arXiv:1911.02116 , 2019
Original
2019
Earlier work this paper cites.
Y. Ren, Y. Ruan, X. Tan, T. Qin, S. Zhao, Z. Zhao, and T.-Y. Liu, “Fastspeech: Fast, robust and controllable text to speech,” Advances in Neural Information Processing Systems , vol. 32, 2019
2019
Earlier work this paper cites.
G. Lample and A. Conneau, “Cross-lingual language model pretraining,” arXiv preprint arXiv:1901.07291 , 2019
Original
2019
Earlier work this paper cites.
D. S. Park, W. Chan, Y. Zhang, C.-C. Chiu, B. Zoph, E. D. Cubuk, and Q. V. Le, “SpecAugment: A simple data augmentation method for automatic speech recognition,” Proc. Interspeech 2019 , pp. 2613–2617, 2019
2019
Earlier work this paper cites.
R. Ardila et al. , “Common Voice: A massively-multilingual speech corpus,” arXiv preprint arXiv:1912.06670 , 2019
Original
2019
Earlier work this paper cites.
F. Wu, A. Fan, A. Baevski, Y. N. Dauphin, and M. Auli, “Pay less attention with lightweight and dynamic convolutions,” arXiv preprint arXiv:1901.10430 , 2019
Original
2019
Earlier work this paper cites.
L. Xue, N. Constant, A. Roberts, M. Kale, R. Al-Rfou, A. Siddhant, A. Barua, and C. Raffel, “mT5: A massively multilingual pre-trained text-to-text transformer,” arXiv preprint arXiv:2010.11934 , 2020
Original
2020
Earlier work this paper cites.

Similar

A. Baevski et al. , “wav2vec 2.0: A framework for self-supervised learning of speech representations,” arXiv preprint arXiv:2006.11477 , 2020
Original
2020
Cited alongside, same era.
Y.-A. Chung, H. Tang, and J. Glass, “Vector-quantized autoregressive predictive coding,” arXiv preprint arXiv:2005.08392 , 2020
Original
2020
Cited alongside, same era.
Y. Zhang et al. , “Pushing the limits of semi-supervised learning for automatic speech recognition,” arXiv preprint arXiv:2010.10504 , 2020
Original
2020
Cited alongside, same era.
A. Gulati, J. Qin, C.-C. Chiu et al. , “Conformer: Convolution-augmented Transformer for speech recognition,” arXiv preprint arXiv:2005.08100 , 2020
Original
2020
Cited alongside, same era.

Then

A. Jaegle, F. Gimeno, A. Brock, O. Vinyals, A. Zisserman, and J. Carreira, “Perceiver: General perception with iterative attention,” in International Conference on Machine Learning . PMLR, 2021, pp. 4651–4664
2021
Later among the works it cites.
S. Ueno, M. Mimura, S. Sakai, and T. Kawahara, “Data augmentation for asr using TTS via a discrete representation,” ASRU , 2021
2021
Later among the works it cites.
I. Elias, H. Zen, J. Shen, Y. Zhang, Y. Jia, R. J. Weiss, and Y. Wu, “Parallel Tacotron: Non-autoregressive and controllable TTS,” in Proc. ICASSP , 2021, pp. 5709–5713
2021
Later among the works it cites.
A. Babu, C. Wang, A. Tjandra, K. Lakhotia, Q. Xu, N. Goyal, K. Singh, P. von Platen, Y. Saraf, J. Pino et al. , “XLS-R: Self-supervised cross-lingual speech representation learning at scale,” arXiv preprint arXiv:2111.09296 , 2021
Original
2021
Later among the works it cites.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

G. Wang, A. Rosenberg, Z. Chen, Y. Zhang, B. Ramabhadran, Y. Wu, and P. Moreno, “Improving speech recognition using consistent predictions on synthesized speech,” in ICASSP , 2020

2020

Cited alongside, same era.

J. Kahn, M. Rivière, W. Zheng et al. , “Libri-Light: A benchmark for asr with limited or no supervision,” in Proc. ICASSP , 2020, pp. 7669–7673

2020

Cited alongside, same era.

V. Pratap, Q. Xu, A. Sriram, G. Synnaeve, and R. Collobert, “MLS: A large-scale multilingual dataset for speech research,” arXiv preprint arXiv:2012.03411 , 2020

2020

Cited alongside, same era.

C. Wang, A. Wu, and J. Pino, “CoVoST 2 and massively multilingual speech-to-text translation,” arXiv preprint arXiv:2007.10310 , 2020

Original

2020

Cited alongside, same era.

J. Ao, R. Wang, L. Zhou, S. Liu, S. Ren, Y. Wu, T. Ko, Q. Li, Y. Zhang, Z. Wei et al. , “SpeechT5: Unified-modal encoder-decoder pre-training for spoken language processing,” arXiv preprint arXiv:2110.07205 , 2021

2021

Cited alongside, same era.

W.-N. Hsu, Y.-H. H. Tsai, B. Bolte, R. Salakhutdinov, and A. Mohamed, “HuBERT: How much can a bad teacher benefit ASR pre-training?” in Proc. ICASSP , 2021, pp. 6533–6537

2021

Cited alongside, same era.

Y.-A. Chung et al. , “W2v-BERT: Combining contrastive learning and masked language modeling for self-supervised speech pre-training,” arXiv preprint arXiv:2108.06209 , 2021

2021

Cited alongside, same era.

A. Bapna, Y.-a. Chung, N. Wu, A. Gulati, Y. Jia, J. H. Clark, M. Johnson, J. Riesa, A. Conneau, and Y. Zhang, “SLAM: A unified encoder for speech and language modeling via speech-text joint pre-training,” arXiv preprint arXiv:2110.10329 , 2021

2021

Cited alongside, same era.

J. Bai, B. Li, Y. Zhang, A. Bapna, and T. N. Sainath, “Joint unsupervised and supervised training for multilingual ASR,” arXiv preprint arXiv:2111.08137 , 2021

2021

Later among the works it cites.

R. Zheng et al. , “Fused acoustic and text encoding for multimodal bilingual pretraining and speech translation,” arXiv preprint arXiv:2102.05766 , 2021

2021

Later among the works it cites.

W. Chan et al. , “SpeechStew: Simply mix all available speech recognition data to train one large neural network,” arXiv preprint arXiv:2104.02133 , 2021

2021

Later among the works it cites.

C. Wang, M. Riviere, A. Lee, A. Wu, C. Talnikar, D. Haziza, M. Williamson, J. Pino, and E. Dupoux, “Voxpopuli: A large-scale multilingual speech corpus for representation learning, semi-supervised learning and interpretation,” arXiv preprint arXiv:2101.00390 , 2021

2021

Later among the works it cites.

J. Valk and T. Alumäe, “VoxLingua107: A dataset for spoken language recognition,” in 2021 IEEE Spoken Language Technology Workshop (SLT) . IEEE, 2021, pp. 652–658

2021

Later among the works it cites.

Y. Tang, C. Tran, X. Li, P.-J. Chen, N. Goyal, V. Chaudhary, J. Gu, and A. Fan, “Multilingual translation from denoising pre-training,” in Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021 , 2021, pp. 3450–3466

2021

Later among the works it cites.

A. Bapna, C. Cherry, Y. Zhang, Y. Jia, M. Johnson, Y. Cheng, S. Khanuja, J. Riesa, and A. Conneau, “mSLAM: Massively multilingual joint pre-training for speech and text,” arXiv preprint arXiv:2202.01374 , 2022

2022

Closest in time.

A. Baevski, W.-N. Hsu, Q. Xu, A. Babu, J. Gu, and M. Auli, “Data2vec: A general framework for self-supervised learning in speech, vision and language,” arXiv preprint arXiv:2202.03555 , 2022

2022

Closest in time.

Z. Chen, Y. Zhang, A. Rosenberg, B. Ramabhadran, P. Moreno, and G. Wang, “Tts4pretrain 2.0: Advancing the use of text and speech in ASR pretraining with consistency and contrastive losses,” in Proc. ICASSP , 2022

2022

Closest in time.