2015

Visually Indicated Sounds

Owens, Andrew, Isola, Phillip, McDermott, Josh et al.

Understand

Objects make distinctive sounds when they are hit or scratched.

These sounds reveal aspects of an object's material properties, as well as the actions that produced them.
In this paper, we propose the task of predicting what sound an object makes when struck as a way of studying physical interactions within a visual scene.
We present an algorithm that synthesizes sound from silent videos of people hitting and scratching objects with a drumstick.

Built on

Can one hear the shape of a drum?
M. Kac · 1966
Earlier work this paper cites.
The estimation of the gradient of a density function, with applications in pattern recognition
K. Fukunaga and L. D. Hostetler · 1975
Earlier work this paper cites.
Derivation of auditory filter shapes from notched-noise data
B. R. Glasberg and B. C. Moore · 1990
Earlier work this paper cites.
What in the world do we hear?: An ecological approach to auditory event perception
W. W. Gaver · 1993
Earlier work this paper cites.
Pattern playback in the 90s
M. Slaney · 1994
Earlier work this paper cites.
Robotic perception of material
E. Krotkov · 1995
Earlier work this paper cites.
Theory of vibration: an introduction
A. A. Shabana · 1995
Earlier work this paper cites.
Long short-term memory
S. Hochreiter and J. Schmidhuber · 1997
Earlier work this paper cites.
Foleyautomatic: physically-based sound effects for interactive simulation and animation
K. Van Den Doel, P. G. Kry, and D. K. Pai · 2001
Earlier work this paper cites.
The acquisition of physical knowledge in infancy: A summary in eight lessons
R. Baillargeon · 2002
Earlier work this paper cites.
Speech enhancement based on wavelet thresholding the multitaper spectrum
Y. Hu and P. C. Loizou · 2004
Earlier work this paper cites.
A tutorial on onset detection in music signals
J. P. Bello, L. Daudet, S. Abdallah, C. Duxbury, M. Davies, and M. B. Sandler · 2005
Earlier work this paper cites.
The development of embodied cognition: Six lessons from babies
L. Smith and M. Gasser · 2005
Earlier work this paper cites.
Statistical modeling of intrinsic structures in impacts sounds
S. Cavaco and M. S. Lewicki · 2007
Earlier work this paper cites.
Human sound source identification
R. A. Lutfi · 2008
Earlier work this paper cites.
Imagenet: A large-scale hierarchical image database
J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei · 2009
Earlier work this paper cites.

Similar

Deep learning from temporal coherence in video
H. Mobahi, R. Collobert, and J. Weston · 2009
Cited alongside, same era.
Interactive learning of the acoustic properties of household objects
J. Sinapov, M. Wiemer, and A. Stoytchev · 2009
Cited alongside, same era.
Sound texture perception via statistics of the auditory periphery: evidence from sound synthesis
J. H. McDermott and E. P. Simoncelli · 2011
Cited alongside, same era.
Multimodal deep learning
J. Ngiam, A. Khosla, M. Kim, J. Nam, H. Lee, and A. Y. Ng · 2011
Cited alongside, same era.
Were those coconuts or horse hoofs? visual context effects on identification and perceived veracity of everyday sounds
T. Bonebright · 2012
Cited alongside, same era.
Spatial pattern of bold fmri activation reveals cross-modal information in auditory cortex
P.-J. Hsieh, J. T. Colas, and N. Kanwisher · 2012
Cited alongside, same era.
Imagenet classification with deep convolutional neural networks
A. Krizhevsky, I. Sutskever, and G. E. Hinton · 2012
Cited alongside, same era.
The origins of inquiry: Inductive inference and exploration in early childhood
L. Schulz · 2012
Cited alongside, same era.
3d convolutional neural networks for human action recognition
S. Ji, W. Xu, M. Yang, and K. Yu · 2013
Cited alongside, same era.
Recognizing materials using perceptually inspired features
L. Sharan, C. Liu, R. Rosenholtz, and E. H. Adelson · 2013
Cited alongside, same era.
Material recognition in the wild with the materials in context database
Original
S. Bell, P. Upchurch, N. Snavely, and K. Bala · 2014
Cited alongside, same era.
The visual microphone: passive recovery of sound from video
A. Davis, M. Rubinstein, N. Wadhwa, G. J. Mysore, F. Durand, and W. T. Freeman · 2014
Cited alongside, same era.

Then

Two-stream convolutional networks for action recognition in videos
K. Simonyan and A. Zisserman · 2014
Later among the works it cites.
Dropout: A simple way to prevent neural networks from overfitting
N. Srivastava, G. Hinton, A. Krizhevsky, I. Sutskever, and R. Salakhutdinov · 2014
Later among the works it cites.
Learning deep features for scene recognition using places database
B. Zhou, A. Lapedriza, J. Xiao, A. Torralba, and A. Oliva · 2014
Later among the works it cites.
Learning to see by moving
P. Agrawal, J. Carreira, and J. Malik · 2015
Closest in time.
Joint object-material category segmentation from audio-visual cues
A. Arnab, M. Sapienza, S. Golodetz, J. Valentin, O. Miksik, S. Izadi, and P. H. S. Torr · 2015
Closest in time.
Visual vibrometry: Estimating material properties from small motion in video
A. Davis, K. L. Bouman, M. Rubinstein, F. Durand, and W. T. Freeman · 2015
Closest in time.
Unsupervised visual representation learning by context prediction
C. Doersch, A. Gupta, and A. A. Efros · 2015
Closest in time.
Long-term recurrent convolutional networks for visual recognition and description
J. Donahue, L. A. Hendricks, S. Guadarrama, M. Rohrbach, S. Venugopalan, K. Saenko, and T. Darrell · 2015
Closest in time.
Unsupervised feature learning from temporal data
Original
R. Goroshin, J. Bruna, J. Tompson, D. Eigen, and Y. LeCun · 2015
Closest in time.
Batch normalization: Accelerating deep network training by reducing internal covariate shift
Original
S. Ioffe and C. Szegedy · 2015
Closest in time.
Learning visual groups from co-occurrences in space and time
Original
P. Isola, D. Zoran, D. Krishnan, and E. H. Adelson · 2015
Closest in time.
Learning image representations tied to ego-motion
D. Jayaraman and K. Grauman · 2015
Closest in time.
Deep learning for acoustic modeling in parametric speech generation: A systematic review of existing techniques and future trends
Z.-H. Ling, S.-Y. Kang, H. Zen, A. Senior, M. Schuster, X.-J. Qian, H. M. Meng, and L. Deng · 2015
Closest in time.
Supersizing self-supervision: Learning to grasp from 50k tries and 700 robot hours
Original
L. Pinto and A. Gupta · 2015
Closest in time.
Anticipating the future by watching unlabeled video
Original
C. Vondrick, H. Pirsiavash, and A. Torralba · 2015
Closest in time.
Unsupervised learning of visual representations using videos
X. Wang and A. Gupta · 2015
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…