2021

SERAB: A multi-lingual benchmark for speech emotion recognition

Scheidwasser-Clow, Neil, Kegler, Mikolaj, Beckmann, Pierre et al.

Understand

Recent developments in speech emotion recognition (SER) often leverage deep neural networks (DNNs).

Comparing and benchmarking different DNN models can often be tedious due to the use of different datasets and evaluation protocols.
To facilitate the process, here, we present the Speech Emotion Recognition Adaptation Benchmark (SERAB), a framework for evaluating the performance and generalization capacity of different approaches for utterance-level SER.
The benchmark is composed of nine datasets for SER in six languages.

Built on

“A database of German emotional speech,”
F. Burkhardt, A. Paeschke, M. Rolfes, W. F. Sendlmeier, and B. Weiss, · 2005
Earlier work this paper cites.
“IEMOCAP: Interactive emotional dyadic motion capture database,”
C. Busso, M. Bulut, C.-C. Lee, A. Kazemzadeh, E. Mower, S. Kim, J. N. Chang, S. Lee, and S. S. Narayanan, · 2008
Earlier work this paper cites.
“Imagenet: A large-scale hierarchical image database,”
J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei, · 2009
Earlier work this paper cites.
“Speaker-dependent audio-visual emotion recognition,”
S. Haq, P. J. Jackson, and J. Edge, · 2009
Earlier work this paper cites.
“openSMILE: the Munich versatile and fast open-source audio feature extractor,”
F. Eyben, M. Wöllmer, and B. Schuller, · 2010
Earlier work this paper cites.
“Automatic stress detection in emergency (telephone) calls,”
I. Lefter, L. J. Rothkrantz, D. A. Van Leeuwen, and P. Wiggers, · 2011
Earlier work this paper cites.
“Scikit-learn: Machine learning in Python,”
F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, et al., · 2011
Earlier work this paper cites.
Computational Paralinguistics: Emotion, Affect and Personality in Speech and Language Processing
B. Schuller and A. Batliner, · 2013
Earlier work this paper cites.
Historical Linguistics: An Introduction
L. Campbell, · 2013
Earlier work this paper cites.
“CREMA-D: Crowd-sourced emotional multimodal actors dataset,”
H. Cao, D. G. Cooper, M. K. Keutmann, R. C. Gur, A. Nenkova, and R. Verma, · 2014
Earlier work this paper cites.
“EMOVO corpus: an Italian emotional speech database,”
G. Costantini, I. Iaderola, A. Paoloni, and M. Todisco, · 2014
Earlier work this paper cites.
“A multi-task learning framework for emotion recognition using 2D continuous space,”
R. Xia and Y. Liu, · 2015
Earlier work this paper cites.

Similar

“Very deep convolutional networks for large-scale image recognition,”
K. Simonyan and A. Zisserman, · 2015
Cited alongside, same era.
“Youtube-8M: A large-scale video classification benchmark,”
Original
S. Abu-El-Haija, N. Kothari, J. Lee, P. Natsev, G. Toderici, B. Varadarajan, and S. Vijayanarasimhan, · 2016
Cited alongside, same era.
“openXBOW: Introducing the Passau open-source crossmodal bag-of-words toolkit,”
M. Schmitt and B. Schuller, · 2017
Cited alongside, same era.
“Audio set: An ontology and human-labeled dataset for audio events,”
J. F. Gemmeke, D. P. Ellis, D. Freedman, A. Jansen, W. Lawrence, R. C. Moore, M. Plakal, and M. Ritter, · 2017
Cited alongside, same era.
“CNN architectures for large-scale audio classification,”
S. Hershey, S. Chaudhuri, D. P. Ellis, J. F. Gemmeke, A. Jansen, R. C. Moore, M. Plakal, D. Platt, R. A. Saurous, B. Seybold, et al., · 2017
Cited alongside, same era.
“MobileNets: Efficient convolutional neural networks for mobile vision applications,”
Original
A. G. Howard, M. Zhu, B. Chen, D. Kalenichenko, W. Wang, T. Weyand, M. Andreetto, and H. Adam, · 2017
Cited alongside, same era.
“Voice stress analysis: A new framework for voice and effort in human performance,”
M. Van Puyvelde, X. Neyt, F. McGlone, and N. Pattyn, · 2018
Cited alongside, same era.
“GLUE: A multi-task benchmark and analysis platform for natural language understanding,”
A. Wang, A. Singh, J. Michael, F. Hill, O. Levy, and S. R. Bowman, · 2018
Cited alongside, same era.
“Speech emotion recognition for performance interaction,”
N. Vryzas, R. Kotsakis, A. Liatsou, C. A. Dimoulas, and G. Kalliris, · 2018
Cited alongside, same era.
“A Canadian French emotional speech dataset,”
P. Gournay, O. Lahaie, and R. Lefebvre, · 2018
Cited alongside, same era.
“The Ryerson Audio-Visual Database of Emotional Speech and Song (RAVDESS): A dynamic, multimodal set of facial and vocal expressions in North American English,”
S. R. Livingstone and F. A. Russo, · 2018
Cited alongside, same era.
“ShEMO: a large-scale validated database for persian speech emotion detection,”
O. M. Nezami, P. J. Lou, and M. Karami, · 2019
Cited alongside, same era.

Then

“A large-scale study of representation learning with the visual task adaptation benchmark,”
Original
X. Zhai, J. Puigcerver, A. Kolesnikov, P. Ruyssen, C. Riquelme, M. Lucic, J. Djolonga, A. S. Pinto, M. Neumann, A. Dosovitskiy, et al., · 2019
Later among the works it cites.
“Towards Learning a Universal Non-Semantic Representation of Speech,”
J. Shor, A. Jansen, R. Maor, O. Lang, O. Tuval, F. C. Quitry, M. Tagliasacchi, I. Shavitt, D. Emanuel, and Y. Haviv, · 2020
Later among the works it cites.
“YAMNet,” https://github.com/tensorflow/models/tree/master/research/audioset/yamnet
M. Plakal and D. Ellis, · 2020
Later among the works it cites.
“Bootstrap your own latent - a new approach to self-supervised learning,”
J.-B. Grill, F. Strub, F. Altché, C. Tallec, P. Richemond, E. Buchatskaya, C. Doersch, B. A. Pires, Z. D. Guo, M. G. Azar, et al., · 2020
Later among the works it cites.
“BYOL for Audio: Self-supervised learning for general-purpose audio representation,”
D. Niizumi, D. Takeuchi, Y. Ohishi, N. Harada, and K. Kashino, · 2021
Closest in time.
“HEAR 2021 NeurIPS Challenge Holistic Evaluation of Audio Representations,” https://neuralaudio.ai/hear2021-holistic-evaluation-of-audio-representations.html
Neural Audio AI, · 2021
Closest in time.
“LSSED: a large-scale dataset and benchmark for speech emotion recognition,”
W. Fan, X. Xu, X. Xing, W. Chen, and D. Huang, · 2021
Closest in time.
“The INTERSPEECH 2021 Computational Paralinguistics Challenge: COVID-19 cough, COVID-19 speech, escalation & primates,”
Original
B. Schuller, A. Batliner, C. Bergler, C. Mascolo, J. Han, I. Lefter, H. Kaya, S. Amiriparian, A. Baird, L. Stappen, et al., · 2021
Closest in time.
“CvT: Introducing convolutions to vision transformers,”
Original
H. Wu, B. Xiao, N. Codella, M. Liu, X. Dai, L. Yuan, and L. Zhang, · 2021
Closest in time.
“On the bias against inductive biases,”
Original
G. Cazenavette and S. Lucey, · 2021
Closest in time.
“An image is worth 16x16 words: Transformers for image recognition at scale,”
A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, et al., · 2021
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…