2021

PSLA: Improving Audio Tagging with Pretraining, Sampling, Labeling, and Aggregation

Gong, Yuan, Chung, Yu-An, Glass, James

Understand

Audio tagging is an active research area and has a wide range of applications.

Since the release of AudioSet, great progress has been made in advancing model performance, which mostly comes from the development of novel model architectures and attention modules.
However, we find that appropriate training techniques are equally important for building audio tagging models with AudioSet, but have not received the attention they deserve.
To fill the gap, in this work, we present PSLA, a collection of training techniques that can noticeably boost the model accuracy including ImageNet pretraining, balanced sampling, data augmentation, label enhancement, model aggregation and their design choices.

Built on

K. Wu and D. G. Childers, “Gender recognition from speech. part i: Coarse analysis,” JASA , vol. 90, no. 4, pp. 1828–1840, 1991
1991
Earlier work this paper cites.
D. G. Childers and K. Wu, “Gender recognition from speech. part ii: Fine analysis,” JASA , vol. 90, no. 4, pp. 1841–1856, 1991
1991
Earlier work this paper cites.
B. Efron and R. J. Tibshirani, An introduction to the bootstrap . CRC press, 1994
1994
Earlier work this paper cites.
L. Breiman, “Bagging predictors,” Machine Learning , vol. 24, no. 2, pp. 123–140, 1996
1996
Earlier work this paper cites.
A. P. Bradley, “The use of the area under the roc curve in the evaluation of machine learning algorithms,” Pattern recognition , vol. 30, no. 7, pp. 1145–1159, 1997
1997
Earlier work this paper cites.
J. Davis and M. Goadrich, “The relationship between precision-recall and roc curves,” in Proceedings of the 23rd international conference on Machine learning , 2006
2006
Earlier work this paper cites.
A. Chandra, H. Chen, and X. Yao, “Trade-off between diversity and accuracy in ensemble generation,” in Multi-objective machine learning . Springer, 2006, pp. 429–464
2006
Earlier work this paper cites.
J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei, “Imagenet: A large-scale hierarchical image database,” in 2009 IEEE conference on computer vision and pattern recognition , 2009, pp. 248–255
2009
Earlier work this paper cites.
H. He and E. A. Garcia, “Learning from imbalanced data,” IEEE Transactions on knowledge and data engineering , vol. 21, no. 9, pp. 1263–1284, 2009
2009
Earlier work this paper cites.
J. Salamon, C. Jacoby, and J. P. Bello, “A dataset and taxonomy for urban sound research,” in Proceedings of the 22nd ACM international conference on Multimedia , 2014, pp. 1041–1044
2014
Earlier work this paper cites.
G. Gwardys and D. M. Grzywczak, “Deep image features in music information retrieval,” International Journal of Electronics and Telecommunications , vol. 60, no. 4, pp. 321–326, 2014
2014
Earlier work this paper cites.
K. J. Piczak, “Esc: Dataset for environmental sound classification,” in Proceedings of the 23rd ACM international conference on Multimedia , 2015, pp. 1015–1018
2015
Earlier work this paper cites.
P. Foster, S. Sigtia, S. Krstulovic, J. Barker, and M. D. Plumbley, “Chime-home: A dataset for sound source recognition in a domestic environment,” in 2015 IEEE Workshop on Applications of Signal Processing to Audio and Acoustics (WASPAA) , 2015, pp. 1–5
2015
Earlier work this paper cites.
D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” in International Conference on Learning Representations , 2015
2015
Earlier work this paper cites.
K. He, X. Zhang, S. Ren, and J. Sun, “Delving deep into rectifiers: Surpassing human-level performance on imagenet classification,” in Proceedings of the IEEE international conference on computer vision , 2015, pp. 1026–1034
2015
Earlier work this paper cites.
C. Szegedy, W. Liu, Y. Jia, P. Sermanet, S. Reed, D. Anguelov, D. Erhan, V. Vanhoucke, and A. Rabinovich, “Going deeper with convolutions,” in Proceedings of the IEEE conference on computer vision and pattern recognition , 2015, pp. 1–9
2015
Earlier work this paper cites.
K. Simonyan and A. Zisserman, “Very deep convolutional networks for large-scale image recognition,” 2015
2015
Earlier work this paper cites.
K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in Proceedings of the IEEE conference on computer vision and pattern recognition , 2016, pp. 770–778
2016
Earlier work this paper cites.
J. F. Gemmeke, D. P. Ellis, D. Freedman, A. Jansen, W. Lawrence, R. C. Moore, M. Plakal, and M. Ritter, “Audio set: An ontology and human-labeled dataset for audio events,” in IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , 2017, pp. 776–780
2017
Earlier work this paper cites.
D. Lee, S. Lee, Y. Han, and K. Lee, “Ensemble of convolutional neural networks for weakly-supervised sound event detection using multiple scale input,” in DCASE Workshop , 2017
2017
Earlier work this paper cites.
A. Mesaros, T. Heittola, A. Diment, B. Elizalde, A. Shah, E. Vincent, B. Raj, and T. Virtanen, “Dcase 2017 challenge setup: Tasks, datasets and baseline system,” in DCASE Workshop , 2017
2017
Earlier work this paper cites.

Similar

Z.-Q. Wang and I. Tashev, “Learning utterance-level representations for speech emotion and age/gender recognition using deep neural networks,” in IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , 2017, pp. 5150–5154
2017
Cited alongside, same era.
G. Huang, Z. Liu, L. Van Der Maaten, and K. Q. Weinberger, “Densely connected convolutional networks,” in Proceedings of the IEEE conference on computer vision and pattern recognition , 2017, pp. 4700–4708
2017
Cited alongside, same era.
Q. Kong, Y. Xu, W. Wang, and M. D. Plumbley, “Audio set classification with attention model: A probabilistic perspective,” in IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , 2018, pp. 316–320
2018
Cited alongside, same era.
C. Yu, K. S. Barsim, Q. Kong, and B. Yang, “Multi-level attention model for weakly supervised audio classification,” in DCASE Workshop , 2018
2018
Cited alongside, same era.
S. Chen, J. Chen, Q. Jin, and A. Hauptmann, “Class-aware self-attention for audio event recognition,” in Proceedings of the 2018 ACM on International Conference on Multimedia Retrieval , 2018, pp. 28–36
2018
Cited alongside, same era.
M. Sandler, A. Howard, M. Zhu, A. Zhmoginov, and L.-C. Chen, “Mobilenetv2: Inverted residuals and linear bottlenecks,” in Proceedings of the IEEE conference on computer vision and pattern recognition , 2018, pp. 4510–4520
2018
Cited alongside, same era.
A. Jansen, M. Plakal, R. Pandya, D. P. Ellis, S. Hershey, J. Liu, R. C. Moore, and R. A. Saurous, “Unsupervised learning of semantic audio representations,” in IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , 2018, pp. 126–130
2018
Cited alongside, same era.
Y. Tokozume, Y. Ushiku, and T. Harada, “Between-class learning for image classification,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , 2018, pp. 5486–5494
2018
Cited alongside, same era.
H. Zhang, M. Cisse, Y. N. Dauphin, and D. Lopez-Paz, “mixup: Beyond empirical risk minimization,” in International Conference on Learning Representations , 2018
2018
Cited alongside, same era.
Y. Tokozume, Y. Ushiku, and T. Harada, “Learning from between-class examples for deep sound recognition,” in International Conference on Learning Representations , 2018
2018
Cited alongside, same era.
A. Shah, A. Kumar, A. G. Hauptmann, and B. Raj, “A closer look at weak label learning for audio events,” arXiv preprint arXiv:1804.09288 , 2018
Original
2018
Cited alongside, same era.
P. Izmailov, D. Podoprikhin, T. Garipov, D. Vetrov, and A. G. Wilson, “Averaging weights leads to wider optima and better generalization,” in 34th Conference on Uncertainty in Artificial Intelligence 2018, UAI 2018 , 2018, pp. 876–885
2018
Cited alongside, same era.

Then

S. Adapa, “Urban sound tagging using convolutional neural networks,” in DCASE Workshop , 2019, p. 5
2019
Later among the works it cites.
D. S. Park, W. Chan, Y. Zhang, C.-C. Chiu, B. Zoph, E. D. Cubuk, and Q. V. Le, “SpecAugment: A simple data augmentation method for automatic speech recognition,” Interspeech , pp. 2613–2617, 2019
2019
Later among the works it cites.
L. H. Ford, “Large-scale acoustic scene analysis with deep residual networks,” Master’s thesis, Massachusetts Institute of Technology, 2019
2019
Later among the works it cites.
M. Meire, L. Vuegen, and P. Karsmakers, “The impact of missing labels and overlapping sound events on multi-label multi-instance learning for sound event classification,” in DCASE Workshop , 2019, pp. 159–163
2019
Later among the works it cites.
B. Athiwaratkun, M. Finzi, P. Izmailov, and A. G. Wilson, “There are many consistent explanations of unlabeled data: Why you should average,” in International Conference on Learning Representations , 2019
2019
Later among the works it cites.
Z. Shi, L. Liu, H. Lin, R. Liu, A. Shi, and S. First, “Hodgepodge: Sound event detection based on ensemble of semi-supervised learning methods,” in DCASE Workshop , 2019, p. 224
2019
Later among the works it cites.
Y. Guo, M. Xu, Z. Wu, J. Wu, and B. Su, “Multi-scale convolutional recurrent neural network with ensemble method for weakly labeled sound event detection,” in 2019 8th International Conference on Affective Computing and Intelligent Interaction Workshops and Demos (ACIIW) , 2019, pp. 1–5
2019
Later among the works it cites.
W. Lim, S. Suh, S. Park, and Y. Jeong, “Sound event detection in domestic environments using ensemble of convolutional recurrent neural networks,” in DCASE Workshop , 2019
2019
Later among the works it cites.
Y. Wang, J. Li, and F. Metze, “A comparison of five multiple instance learning pooling functions for sound event detection with weak labeling,” in IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , 2019, pp. 31–35
2019
Later among the works it cites.
Q. Kong, Y. Cao, T. Iqbal, Y. Wang, W. Wang, and M. D. Plumbley, “Panns: Large-scale pretrained audio neural networks for audio pattern recognition,” IEEE/ACM Transactions on Audio, Speech, and Language Processing , vol. 28, pp. 2880–2894, 2020
2020
Later among the works it cites.
E. Fonseca, X. Favory, J. Pons, F. Font, and X. Serra, “Fsd50k: an open dataset of human-labeled sound events,” arXiv preprint arXiv:2010.00475 , 2020
Original
2020
Later among the works it cites.
K. Palanisamy, D. Singhania, and A. Yao, “Rethinking cnn models for audio classification,” arXiv preprint arXiv:2007.11154 , 2020
Original
2020
Later among the works it cites.
E. Fonseca, S. Hershey, M. Plakal, D. P. Ellis, A. Jansen, and R. C. Moore, “Addressing missing labels in large-scale sound event recognition using a teacher-student framework with loss masking,” IEEE Signal Processing Letters , vol. 27, pp. 1235–1239, 2020
2020
Later among the works it cites.
P. Lopez-Meyer, J. A. del Hoyo Ontiveros, G. Stemmer, L. Nachman, and J. Huang, “Ensemble of convolutional neural networks for the DCASE 2020 acoustic scene classification challenge,” in DCASE Workshop , 2020
2020
Later among the works it cites.
A. Saeed, D. Grangier, and N. Zeghidour, “Contrastive learning of general-purpose audio representations,” arXiv preprint arXiv:2010.10915 , 2020
Original
2020
Later among the works it cites.
J. Shor, A. Jansen, R. Maor, O. Lang, O. Tuval, F. de Chaumont Quitry, M. Tagliasacchi, I. Shavitt, D. Emanuel, and Y. Haviv, “Towards learning a universal non-semantic representation of speech,” Interspeech , pp. 140–144, 2020
2020
Later among the works it cites.
M. Tagliasacchi, B. Gfeller, F. de Chaumont Quitry, and D. Roblek, “Pre-training audio representations with self-supervision,” IEEE Signal Processing Letters , vol. 27, pp. 600–604, 2020
2020
Later among the works it cites.
L. Wang, K. Kawakami, and A. van den Oord, “Contrastive predictive coding of audio with an adversary,” Interspeech , pp. 826–830, 2020
2020
Later among the works it cites.
A. Guzhov, F. Raue, J. Hees, and A. Dengel, “Esresnet: Environmental sound classification based on visual domain models,” in 2020 25th International Conference on Pattern Recognition (ICPR) , 2021, pp. 4933–4940
2021
Closest in time.
P. Verma and J. Berger, “Audio transformers: Transformer architectures for large scale audio understanding. adieu convolutions,” arXiv preprint arXiv:2105.00335 , 2021
Original
2021
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…