2016

SoundNet: Learning Sound Representations from Unlabeled Video

Aytar, Yusuf, Vondrick, Carl, Torralba, Antonio

Understand

We learn rich natural sound representations by capitalizing on large amounts of unlabeled sound data collected in the wild.

We leverage the natural synchronization between vision and sound to learn an acoustic representation using two-million unlabeled videos.
Unlabeled video has the advantage that it can be economically acquired at massive scales, yet contains useful signals about natural sound.
We propose a student-teacher training procedure which transfers discriminative visual knowledge from well established visual recognition models into the sound modality using unlabeled video as a bridge.

Built on

Gradient-based learning applied to document recognition
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner · 1998
Earlier work this paper cites.
Visualizing data using t-sne
Laurens Van der Maaten and Geoffrey Hinton · 2008
Earlier work this paper cites.
Unsupervised feature learning for audio classification using convolutional deep belief networks
Honglak Lee, Peter Pham, Yan Largman, and Andrew Y Ng · 2009
Earlier work this paper cites.
Tabula rasa: Model transfer for object category detection
Yusuf Aytar and Andrew Zisserman · 2011
Earlier work this paper cites.
The million song dataset
Thierry Bertin-Mahieux, Daniel PW Ellis, Brian Whitman, and Paul Lamere · 2011
Earlier work this paper cites.
Multimodal deep learning
Jiquan Ngiam, Aditya Khosla, Mingyu Kim, Juhan Nam, Honglak Lee, and Andrew Y Ng · 2011
Earlier work this paper cites.
Imagenet classification with deep convolutional neural networks
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton · 2012
Earlier work this paper cites.
Figure-ground segmentation by transferring window masks
Daniel Kuettel and Vittorio Ferrari · 2012
Earlier work this paper cites.
Watching unlabeled video helps learn new human actions from very few labeled snapshots
Chao-Yeh Chen and Kristen Grauman · 2013
Earlier work this paper cites.
Audio-visual deep learning for noise robust speech recognition
Jing Huang and Brian Kingsbury · 2013
Earlier work this paper cites.
Auditory scene classification using machine learning techniques
David Li, Jason Tam, and Derek Toub · 2013
Earlier work this paper cites.
Recurrence quantification analysis features for environmental sound recognition
Guido Roma, Waldo Nogueira, and Perfecto Herrera · 2013
Earlier work this paper cites.
Deep content-based music recommendation
Aaron Van den Oord, Sander Dieleman, and Benjamin Schrauwen · 2013
Earlier work this paper cites.
Do deep nets really need to be deep?
Jimmy Ba and Rich Caruana · 2014
Earlier work this paper cites.

Similar

Deep speech: Scaling up end-to-end speech recognition
Original
Awni Hannun, Carl Case, Jared Casper, Bryan Catanzaro, Greg Diamos, Erich Elsen, Ryan Prenger, Sanjeev Satheesh, Shubho Sengupta, Adam Coates, et al · 2014
Cited alongside, same era.
Adam: A method for stochastic optimization
Original
Diederik Kingma and Jimmy Ba · 2014
Cited alongside, same era.
Very deep convolutional networks for large-scale image recognition
Original
Karen Simonyan and Andrew Zisserman · 2014
Cited alongside, same era.
Sequence to sequence learning with neural networks
Ilya Sutskever, Oriol Vinyals, and Quoc V Le · 2014
Cited alongside, same era.
Learning deep features for scene recognition using places database
Bolei Zhou, Agata Lapedriza, Jianxiong Xiao, Antonio Torralba, and Aude Oliva · 2014
Cited alongside, same era.
Part level transfer regularization for enhancing exemplar svms
Yusuf Aytar and Andrew Zisserman · 2015
Cited alongside, same era.
Acoustic scene classification: Classifying environments from the sounds they produce
Daniele Barchiesi, Dimitrios Giannoulis, Dan Stowell, and Mark D Plumbley · 2015
Cited alongside, same era.
Polyphonic sound event detection using multi label deep neural networks
Emre Cakir, Toni Heittola, Heikki Huttunen, and Tuomas Virtanen · 2015
Cited alongside, same era.
Cross modal distillation for supervision transfer
Original
Saurabh Gupta, Judy Hoffman, and Jitendra Malik · 2015
Cited alongside, same era.
Distilling the knowledge in a neural network
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean · 2015
Cited alongside, same era.
Batch normalization: Accelerating deep network training by reducing internal covariate shift
Sergey Ioffe and Christian Szegedy · 2015
Cited alongside, same era.
Fully convolutional networks for semantic segmentation
Jonathan Long, Evan Shelhamer, and Trevor Darrell · 2015
Cited alongside, same era.

Then

Environmental sound classification with convolutional neural networks
Karol J Piczak · 2015
Later among the works it cites.
Esc: Dataset for environmental sound classification
Karol J Piczak · 2015
Later among the works it cites.
Histogram of gradients of time-frequency representations for audio scene classification
Alain Rakotomamonjy and Gilles Gasso · 2015
Later among the works it cites.
Imagenet large scale visual recognition challenge
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al · 2015
Later among the works it cites.
Unsupervised feature learning for urban sound classification
Justin Salamon and Juan Pablo Bello · 2015
Later among the works it cites.
Detection and classification of acoustic scenes and events
Dan Stowell, Dimitrios Giannoulis, Emmanouil Benetos, Mathieu Lagrange, and Mark D Plumbley · 2015
Later among the works it cites.
Dense optical flow prediction from a static image
Jacob Walker, Abhinav Gupta, and Martial Hebert · 2015
Later among the works it cites.
Learning aligned cross-modal representations from weakly aligned data
Lluis Castrejon, Yusuf Aytar, Carl Vondrick, Hamed Pirsiavash, and Antonio Torralba · 2016
Closest in time.
Cnn architectures for large-scale audio classification
Shawn Hershey, Sourish Chaudhuri, Daniel PW Ellis, Jort F Gemmeke, Aren Jansen, R Channing Moore, Manoj Plakal, Devin Platt, Rif A Saurous, Bryan Seybold, et al · 2016
Closest in time.
Comparing time and frequency domain for audio event recognition using deep learning
Lars Hertel, Huy Phan, and Alfred Mertins · 2016
Closest in time.
The open world of micro-videos
Phuc Xuan Nguyen, Gregory Rogez, Charless Fowlkes, and Deva Ramamnan · 2016
Closest in time.
Yfcc100m: The new data in multimedia research
Bart Thomee, David A Shamma, Gerald Friedland, Benjamin Elizalde, Karl Ni, Douglas Poland, Damian Borth, and Li-Jia Li · 2016
Closest in time.
Anticipating visual representations from unlabeled video
Carl Vondrick, Hamed Pirsiavash, and Antonio Torralba · 2016
Closest in time.
Generating videos with scene dynamics
Carl Vondrick, Hamed Pirsiavash, and Antonio Torralba · 2016
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…