2021

Quantifying Bias in Automatic Speech Recognition

Feng, Siyuan, Kudina, Olya, Halpern, Bence Mark et al.

Understand

Automatic speech recognition (ASR) systems promise to deliver objective interpretation of human speech.

Practice and recent evidence suggests that the state-of-the-art (SotA) ASRs struggle with the large variation in speech due to e.g., gender, age, speech impairment, race, and accents.
Many factors can cause the bias of an ASR system.
Our overarching goal is to uncover bias in ASR systems to work towards proactive bias mitigation in ASR.

Built on

N. Oostdijk, “The spoken Dutch corpus. overview and first evaluation.” in LREC . Athens, Greece, 2000, pp. 887–894
2000
Earlier work this paper cites.
M. Adda-Decker and L. Lamel, “Do speech recognizers prefer female speakers?” in Proc. INTERSPEECH , 2005
2005
Earlier work this paper cites.
M. Schuster, A. Maier, T. Haderlein, E. Nkenke, U. Wohlleben, F. Rosanowski, U. Eysholdt, and E. Nöth, “Evaluation of speech intelligibility for children with cleft lip and palate by means of automatic speech recognition,” International Journal of Pediatric Otorhinolaryngology , vol. 70, no. 10, pp. 1741–1747, 2006
2006
Earlier work this paper cites.
C. Cucchiarini, O. van Herwijnen, F. Smits et al. , “JASMIN-CGN: Extension of the spoken dutch corpus with speech of elderly people, children and non-natives in the human-machine interaction modality,” in Proc. LREC , 2006
2006
Earlier work this paper cites.
D. A. v. Leeuwen, J. Kessens, E. Sanders, and H. v. d. Heuvel, “Results of the n-best 2008 dutch speech recognition evaluation,” in Proc. INTERSPEECH , 2009
2009
Earlier work this paper cites.
S. Goldwater, D. Jurafsky, and C. D. Manning, “Which words are hard to recognize? prosodic, lexical, and disfluency factors that increase speech recognition error rates,” Speech Communication , vol. 52, no. 3, pp. 181–200, 2010
2010
Earlier work this paper cites.
G. E. Dahl, D. Yu, L. Deng, and A. Acero, “Context-dependent pre-trained deep neural networks for large-vocabulary speech recognition,” IEEE Transactions on audio, speech, and language processing , vol. 20, no. 1, pp. 30–42, 2011
2011
Earlier work this paper cites.
D. Povey, A. Ghoshal, G. Boulianne, L. Burget, O. Glembek, N. Goel, M. Hannemann, P. Motlicek, Y. Qian, P. Schwarz et al. , “The Kaldi speech recognition toolkit,” in Proc. ASRU , 2011
2011
Earlier work this paper cites.
M. Abu Shariah and M. Sawalha, “The effects of speakers’ gender, age, and region on overall performance of arabic automatic speech recognition systems using the phonetically rich and balanced modern standard arabic speech corpus,” in Proceedings of the 2nd Workshop of Arabic Corpus Linguistics WACL-2 , 2013
2013
Earlier work this paper cites.

Similar

T. Ko, V. Peddinti, D. Povey, and S. Khudanpur, “Audio augmentation for speech recognition.” in Proc. INTERSPEECH , 2015, pp. 3586–3589
2015
Cited alongside, same era.
D. Snyder, G. Chen, and D. Povey, “Musan: A music, speech, and noise corpus,” arXiv preprint arXiv:1510.08484 , 2015
Original
2015
Cited alongside, same era.
D. Povey, V. Peddinti, D. Galvez, P. Ghahremani, V. Manohar, X. Na, Y. Wang, and S. Khudanpur, “Purely sequence-trained neural networks for ASR based on lattice-free MMI,” in Proc. INTERSPEECH , 2016, pp. 2751–2755
2016
Cited alongside, same era.
R. Tatman, “Gender and dialect bias in youtube’s automatic captions,” in Proceedings of the First ACL Workshop on Ethics in Natural Language Processing , 2017, pp. 53–59
2017
Cited alongside, same era.
R. Tatman and C. Kasten, “Effects of talker dialect, gender & race on accuracy of bing speech and youtube automatic captions.” in Proc. INTERSPEECH , 2017, pp. 934–938
2017
Cited alongside, same era.
Y. Qian, K. Evanini, X. Wang, C. M. Lee, and M. Mulholland, “Bidirectional lstm-rnn for improving automated assessment of non-native children’s speech.” in INTERSPEECH , 2017, pp. 1417–1421
2017
Cited alongside, same era.
A. Caliskan, J. J. Bryson, and A. Narayanan, “Semantics derived automatically from language corpora contain human-like biases,” Science , vol. 356, no. 6334, pp. 183–186, 2017
2017
Cited alongside, same era.
T. Ko, V. Peddinti, D. Povey, M. L. Seltzer, and S. Khudanpur, “A study on data augmentation of reverberant speech for robust speech recognition,” in Proc. ICASSP , 2017, pp. 5220–5224
2017
Cited alongside, same era.

Then

H. Xu, K. Li, Y. Wang, J. Wang, S. Kang, X. Chen, D. Povey, and S. Khudanpur, “Neural network language modeling with letter-based features and importance sampling,” in Proc. ICASSP , 2018, pp. 6109–6113
2018
Later among the works it cites.
M. Garnerin, S. Rossato, and L. Besacier, “Gender representation in french broadcast corpora and its impact on asr performance,” in Proceedings of the 1st International Workshop on AI for Smart TV Content Production, Access and Delivery , 2019, pp. 3–9
2019
Later among the works it cites.
L. Moro-Velázquez, J. Cho, S. Watanabe, M. A. Hasegawa-Johnson, O. Scharenborg, H. Kim, and N. Dehak, “Study of the performance of automatic speech recognition systems in speakers with parkinson’s disease,” in Proc. INTERSPEECH , 2019, pp. 3875–3879
2019
Later among the works it cites.
A. Palanica, A. Thommandram, A. Lee, M. Li, and Y. Fossat, “Do you understand the words that are comin outta my mouth? voice assistant comprehension of medication names,” NPJ digital medicine , vol. 2, no. 1, pp. 1–6, 2019
2019
Later among the works it cites.
A. Koenecke, A. Nam, E. Lake, J. Nudell, M. Quartey, Z. Mengesha, C. Toups, J. R. Rickford, D. Jurafsky, and S. Goel, “Racial disparities in automated speech recognition,” Proceedings of the National Academy of Sciences , vol. 117, no. 14, pp. 7684–7689, 2020
2020
Later among the works it cites.
B. M. Halpern, R. van Son, M. W. M. van den Brekel, and O. Scharenborg, “Detecting and analysing spontaneous oral cancer speech in the wild,” in Proc. INTERSPEECH , 2020, pp. 4826–4830
2020
Later among the works it cites.
Y. Wu, D. Rough, A. Bleakley, J. Edwards, O. Cooney, P. R. Doyle, L. Clark, and B. R. Cowan, “See what i’m saying? comparing intelligent personal assistant use for native and non-native language speakers,” in 22nd International Conference on Human-Computer Interaction with Mobile Devices and Services , 2020, pp. 1–9
2020
Later among the works it cites.
O. Kudina and B. de Boer, “Co-designing diagnosis: Towards a responsible integration of machine learning decision-support systems in medical diagnostics,” Journal of Evaluation in Clinical Practice , 2021
Original
2021
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…