2021

Timers and Such: A Practical Benchmark for Spoken Language Understanding with Numbers

Lugosch, Loren, Papreja, Piyush, Ravanelli, Mirco et al.

Understand

This paper introduces Timers and Such, a new open source dataset of spoken English commands for common voice control use cases involving numbers.

We describe the gap in existing spoken language understanding datasets that Timers and Such fills, the design and creation of the dataset, and experiments with a number of ASR-based and end-to-end baseline models, the code for which has been made available as part of the SpeechBrain toolkit.

Built on

Y.-A. Chung, C. Zhu, and M. Zeng, “SPLAT: Speech-Language Joint Pre-Training for Spoken Language Understanding,” in Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies , 2021, pp. 1897–1907
1907
Earlier work this paper cites.
R. G. Leonard and G. Doddington, “Tidigits ldc93s10,” Web Download. Philadelphia: Linguistic Data Consortium , 1993
1993
Earlier work this paper cites.
J. S. Garofolo, L. F. Lamel, W. M. Fisher, J. G. Fiscus, D. S. Pallett, and N. L. Dahlgren, “DARPA TIMIT Acoustic Phonetic Continuous Speech Corpus CDROM,” 1993
1993
Earlier work this paper cites.
D. Hakkani-Tür, F. Béchet, G. Riccardi, and G. Tur, “Beyond ASR 1-best: Using word confusion networks in spoken language understanding,” Computer Speech & Language , vol. 20, no. 4, pp. 495–514, 2006
2006
Earlier work this paper cites.
A. Graves, S. Fernández, F. Gomez, and J. Schmidhuber, “Connectionist Temporal Classification: Labelling Unsegmented Sequence Data with Recurrent Neural Networks,” ICML , 2006
2006
Earlier work this paper cites.
A. Graves, “Sequence transduction with recurrent neural networks,” ICML Workshop on Representation Learning , 2012
2012
Earlier work this paper cites.
V. Panayotov, G. Chen, D. Povey, and S. Khudanpur, “LibriSpeech: an ASR corpus based on public domain audio books,” ICASSP , 2015
2015
Earlier work this paper cites.
D. Bahdanau, K. Cho, and Y. Bengio, “Neural machine translation by jointly learning to align and translate,” ICLR , 2015
2015
Earlier work this paper cites.
J. Chorowski, D. Bahdanau, D. Serdyuk, K. Cho, and Y. Bengio, “Attention-based models for speech recognition,” NeurIPS , 2015
2015
Earlier work this paper cites.
W. Chan, N. Jaitly, Q. Le, and O. Vinyals, “Listen, attend and spell: A neural network for large vocabulary conversational speech recognition,” in 2016 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) . IEEE, 2016, pp. 4960–4964
2016
Earlier work this paper cites.
C. Veaux, J. Yamagishi, K. MacDonald et al. , “CSTR VCTK corpus: English multi-speaker corpus for CSTR voice cloning toolkit,” University of Edinburgh. The Centre for Speech Technology Research (CSTR) , 2017
2017
Earlier work this paper cites.
S. Kim, T. Hori, and S. Watanabe, “Joint CTC-attention based end-to-end speech recognition using multi-task learning,” in 2017 IEEE international conference on acoustics, speech and signal processing (ICASSP) . IEEE, 2017, pp. 4835–4839
2017
Earlier work this paper cites.
D. Serdyuk, Y. Wang, C. Fuegen, A. Kumar, B. Liu, and Y. Bengio, “Towards end-to-end spoken language understanding,” ICASSP , 2018
2018
Earlier work this paper cites.

Similar

P. Warden, “Speech commands: A dataset for limited-vocabulary speech recognition,” arXiv preprint arXiv:1804.03209 , 2018
Original
2018
Cited alongside, same era.
Z. Jackson, C. Souza, J. Flaks, Y. Pan, H. Nicolas, and A. Thite, “Jakobovski/free-spoken-digit-dataset: v1.0.8,” 2018. [Online]. Available: https://doi.org/10.5281/zenodo.1342401
2018
Cited alongside, same era.
Y. Taigman, L. Wolf, A. Polyak, and E. Nachmani, “VoiceLoop: Voice fitting and synthesis via a phonological loop,” ICLR , 2018
2018
Cited alongside, same era.
T. Kudo and J. Richardson, “SentencePiece: A simple and language independent subword tokenizer and detokenizer for neural text processing,” arXiv preprint arXiv:1808.06226 , 2018
Original
2018
Cited alongside, same era.
A. Kannan, Y. Wu, P. Nguyen, T. N. Sainath, Z. Chen, and R. Prabhavalkar, “An analysis of incorporating an external language model into a sequence-to-sequence model,” in 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) . IEEE, 2018, pp. 1–5828
2018
Cited alongside, same era.
P. Haghani, A. Narayanan, M. Bacchiani, G. Chuang, N. Gaur, P. Moreno, R. Prabhavalkar, Z. Qu, and A. Waters, “From Audio to Semantics: Approaches to end-to-end spoken language understanding,” IEEE Spoken Language Technology Workshop (SLT) , 2018
2018
Cited alongside, same era.
Y. Jia, Y. Zhang, R. J. Weiss, Q. Wang, J. Shen, F. Ren, Z. Chen, P. Nguyen, R. Pang, I. L. Moreno et al. , “Transfer learning from speaker verification to multispeaker text-to-speech synthesis,” NeurIPS , 2018
2018
Cited alongside, same era.
A. Saade, A. Coucke, A. Caulier, J. Dureau, A. Ball, T. Bluche, D. Leroy, C. Doumouro, T. Gisselbrecht, F. Caltagirone, T. Lavril, and M. Primet, “Spoken language understanding on the edge,” NeurIPS Workshop on Energy Efficient Machine Learning and Cognitive Computing , 2019
2019
Cited alongside, same era.
L. Lugosch, M. Ravanelli, P. Ignoto, V. S. Tomar, and Y. Bengio, “Speech model pre-training for end-to-end spoken language understanding,” Interspeech , 2019
2019
Cited alongside, same era.
S. Pascual, M. Ravanelli, J. Serra, A. Bonafonte, and Y. Bengio, “Learning problem-agnostic speech representations from multiple self-supervised tasks,” Interspeech , 2019
2019
Cited alongside, same era.
A. Ovadya and J. Whittlestone, “Reducing malicious use of synthetic media research: Considerations and potential release practices for machine learning,” arXiv preprint arXiv:1907.11274 , 2019
Original
2019
Cited alongside, same era.
C. Jemine, “Master thesis: Real-time voice cloning,” 2019
2019
Cited alongside, same era.

Then

A. Baevski, H. Zhou, A. Mohamed, and M. Auli, “wav2vec 2.0: A framework for self-supervised learning of speech representations,” NeurIPS , 2020
2020
Later among the works it cites.
R. Ardila, M. Branson, K. Davis, M. Henretty, M. Kohler, J. Meyer, R. Morais, L. Saunders, F. M. Tyers, and G. Weber, “Common Voice: A massively-multilingual speech corpus,” LREC , 2020
2020
Later among the works it cites.
L. Lugosch, B. H. Meyer, D. Nowrouzezahrai, and M. Ravanelli, “Using speech synthesis to train end-to-end spoken language understanding models,” in IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) . IEEE, 2020, pp. 8499–8503
2020
Later among the works it cites.
P. Wang, L. Wei, Y. Cao, J. Xie, and Z. Nie, “Large-scale unsupervised pre-training for end-to-end spoken language understanding,” in ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) . IEEE, 2020, pp. 7999–8003
2020
Later among the works it cites.
Y. Huang, H.-K. Kuo, S. Thomas, Z. Kons, K. Audhkhasi, B. Kingsbury, R. Hoory, and M. Picheny, “Leveraging unpaired text data for training end-to-end speech-to-intent systems,” in ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) . IEEE, 2020, pp. 7984–7988
2020
Later among the works it cites.
T. Likhomanenko, Q. Xu, V. Pratap, P. Tomasello, J. Kahn, G. Avidov, R. Collobert, and G. Synnaeve, “Rethinking Evaluation in ASR: Are Our Models Robust Enough?” arXiv preprint arXiv:2010.11745 , 2020
Original
2020
Later among the works it cites.
M. Mhiri, S. Myer, and V. S. Tomar, “A low latency ASR-free end to end spoken language understanding system,” Interspeech , 2020
2020
Later among the works it cites.
A. Tamkin, M. Wu, and N. Goodman, “Viewmaker networks: Learning views for unsupervised representation learning,” ICLR , 2021
2021
Closest in time.
Y.-A. Chung, C. Zhu, and M. Zeng, “Semi-supervised speech-language joint pre-training for spoken language understanding,” NAACL , 2021
2021
Closest in time.
S. Seo, D. Kwak, and B. Lee, “Integration of pre-trained networks with continuous token interface for end-to-end spoken language understanding,” arXiv:2104.07253 , 2021
Original
2021
Closest in time.
M. Ravanelli, T. Parcollet, P. Plantinga, A. Rouhe, S. Cornell, L. Lugosch, C. Subakan, N. Dawalatabad, A. Heba, J. Zhong, J.-C. Chou, S.-L. Yeh, S.-W. Fu, C.-F. Liao, E. Rastorgueva, F. Grondin, W. Aris, H. Na, Y. Gao, R. D. Mori, and Y. Bengio, “SpeechBrain: A general-purpose speech toolkit,” 2021, arXiv:2106.04624
Original
2021
Closest in time.
B. Sharma, M. Madhavi, and H. Li, “Leveraging acoustic and linguistic embeddings from pretrained speech and language models for intent classification,” in ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) . IEEE, 2021, pp. 7498–7502
2021
Closest in time.
N. Potdar, A. R. Avila, C. Xing, D. Wang, Y. Cao, and X. Chen, “A streaming end-to-end framework for spoken language understanding,” IJCAI , 2021
2021
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…