2021

Towards Learning Universal Audio Representations

Wang, Luyu, Luc, Pauline, Wu, Yan et al.

Understand

The ability to learn universal audio representations that can solve diverse speech, music, and environment tasks can spur many applications that require general sound content understanding.

In this work, we introduce a holistic audio representation evaluation suite (HARES) spanning 12 downstream tasks across audio domains and provide a thorough empirical study of recent sound representation learning systems on that benchmark.
We discover that previous sound event classification or speech models do not generalize outside of their domains.
We observe that more robust audio representations can be learned with the SimCLR objective; however, the model's transferability depends heavily on the model architecture.

Built on

“Evaluation of algorithms using games: The case of music tagging.,”
E Law, K West, M Mandel, M Bay, and S Downie, · 2009
Earlier work this paper cites.
“Scikit-learn: Machine learning in python,”
F Pedregosa, G Varoquaux, A Gramfort, V Michel, B Thirion, O Grisel, M Blondel, P Prettenhofer, R Weiss, V Dubourg, J Vanderplas, A Passos, and Cournapeau D, · 2011
Earlier work this paper cites.
“Adam: A method for stochastic optimization,”
Original
D Kingma and J Ba, · 2014
Earlier work this paper cites.
“Dropout: a simple way to prevent neural networks from overfitting,”
N Srivastava, G Hinton, A Krizhevsky, I Sutskever, and R Salakhutdinov, · 2014
Earlier work this paper cites.
“ESC: Dataset for environmental sound classification,”
K Piczak, · 2015
Earlier work this paper cites.
“Batch normalization: Accelerating deep network training by reducing internal covariate shift,”
S Ioffe and C Szegedy, · 2015
Earlier work this paper cites.
“Librispeech: an asr corpus based on public domain audio books,”
V Panayotov, G Chen, D Povey, and S Khudanpur, · 2015
Earlier work this paper cites.
“Layer normalization,”
Original
J Ba, J Kiros, and G Hinton, · 2016
Earlier work this paper cites.
“Instance normalization: The missing ingredient for fast stylization,”
Original
D Ulyanov, A Vedaldi, and V Lempitsky, · 2016
Earlier work this paper cites.
“Deep networks with stochastic depth,”
G Huang, Y Sun, Z Liu, D Sedra, and K Weinberger, · 2016
Earlier work this paper cites.
“Audio Set: An ontology and human-labeled dataset for audio events,”
J Gemmeke, D Ellis, D Freedman, A Jansen, W Lawrence, C Moore, M Plakal, and M Ritter, · 2017
Earlier work this paper cites.
“Neural audio synthesis of musical notes with WaveNet autoencoders,”
J Engel, C Resnick, A Roberts, S Dieleman, M Norouzi, D Eck, and K Simonyan, · 2017
Earlier work this paper cites.
“SGDR: Stochastic gradient descent with warm restarts,”
I Loshchilov and F Hutter, · 2017
Earlier work this paper cites.
“VoxCeleb: a large-scale speaker identification dataset,”
Original
A Nagrani, J Chung, and A Zisserman, · 2017
Earlier work this paper cites.

Similar

“End-to-end learning for music audio tagging at scale,”
Original
J Pons, O Nieto, M Prockup, E Schmidt, A Ehmann, and X Serra, · 2017
Cited alongside, same era.
“Glue: A multi-task benchmark and analysis platform for natural language understanding,”
Original
A Wang, A Singh, J Michael, F Hill, O Levy, and S Bowman, · 2018
Cited alongside, same era.
“Unsupervised learning of semantic audio representations,”
A Jansen, M Plakal, R Pandya, D Ellis, S Hershey, J Liu, C Moore, and R Saurous, · 2018
Cited alongside, same era.
“Representation learning with contrastive predictive coding,”
Original
A van den Oord, Y Li, and O Vinyals, · 2018
Cited alongside, same era.
“TUT urban acoustic scenes 2018,”
T Heittola, A Mesaros, and T Virtanen, · 2018
Cited alongside, same era.
“Speech Commands: A dataset for limited-vocabulary speech recognition,”
Original
P Warden, · 2018
Cited alongside, same era.
“VoxForge,”
K MacLean, · 2018
Cited alongside, same era.
“PANNs: Large-scale pretrained audio neural networks for audio pattern recognition,”
Original
Q Kong, Y Cao, T Iqbal, Y Wang, W Wang, and M Plumbley, · 2019
Cited alongside, same era.
“wav2vec: Unsupervised pre-training for speech recognition,”
S Schneider, A Baevski, R Collobert, and M Auli, · 2019
Cited alongside, same era.
“Speech model pre-training for end-to-end spoken language understanding,”
Original
L Lugosch, M Ravanelli, P Ignoto, V Tomar, and Y Bengio, · 2019
Cited alongside, same era.
“Automatic acoustic detection of birds through deep learning: the first bird audio detection challenge,”
D Stowell, M Wood, H Pamuła, Y Stylianou, and H Glotin, · 2019
Cited alongside, same era.
“Weight standardization,”
Original
S Qiao, H Wang, C Liu, W Shen, and A Yuille, · 2019
Cited alongside, same era.

Then

“Contrastive predictive coding of audio with an adversary,”
L Wang, K Kawakami, and A van den Oord, · 2020
Later among the works it cites.
“Multi-format contrastive learning of audio representations,”
L Wang and A van den Oord, · 2020
Later among the works it cites.
“Pre-training audio representations with self-supervision,”
M Tagliasacchi, B Gfeller, F de Chaumont Quitry, and D Roblek, · 2020
Later among the works it cites.
“Contrastive learning of general-purpose audio representations,”
Original
A Saeed, D Grangier, and N Zeghidour, · 2020
Later among the works it cites.
“SUPERB: Speech processing universal performance benchmark,”
Original
S Yang, P Chi, Y Chuang, C Lai, K Lakhotia, Y Lin, A Liu, J Shi, X Chang, G Lin, et al., · 2021
Closest in time.
“AST: Audio spectrogram transformer,”
Original
Y Gong, Y Chung, and J Glass, · 2021
Closest in time.
“BYOL for audio: Self-supervised learning for general-purpose audio representation,”
Original
D Niizumi, D Takeuchi, Y Ohishi, N Harada, and K Kashino, · 2021
Closest in time.
“Slow-fast auditory streams for audio recognition,”
E Kazakos, A Nagrani, A Zisserman, and D Damen, · 2021
Closest in time.
“High-performance large-scale image recognition without normalization,”
Original
A Brock, S De, S L Smith, and K Simonyan, · 2021
Closest in time.
“Contrastive learning of musical representations,”
Original
J Spijkervet and J Burgoyne, · 2021
Closest in time.
“Multimodal self-supervised learning of general audio representations,”
Original
L Wang, P Luc, A Recasens, J Alayrac, and A van den Oord, · 2021
Closest in time.
“Characterizing signal propagation to close the performance gap in unnormalized resnets,”
Original
A Brock, S De, and S Smith, · 2021
Closest in time.
“Broaden your views for self-supervised video learning,”
Original
A Recasens, P Luc, J Alayrac, L Wang, F Strub, C Tallec, M Malinowski, V Patraucean, F Altché, M Valko, J Grill, A van den Oord, and A Zisserman, · 2021
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…