2015

Learning image representations tied to ego-motion

Jayaraman, Dinesh, Grauman, Kristen

Understand

Understanding how images of objects and scenes behave in response to specific ego-motions is a crucial aspect of proper visual development, yet existing visual learning methods are conspicuously disconnected from the physical source of their images.

We propose to exploit proprioceptive motor signals to provide unsupervised regularization in convolutional neural networks to learn visual representations from egocentric video.
Specifically, we enforce that our learned features exhibit equivariance i.e.
they respond predictably to transformations associated with distinct ego-motions.

Built on

Movement-produced stimulation in the development of visually guided behavior
R. Held and A. Hein · 1963
Earlier work this paper cites.
Signature verification using a “Siamese” time delay neural network
J. Bromley, J. W. Bentz, L. Bottou, I. Guyon, Y. LeCun, C. Moore, E. Säckinger, and R. Shah · 1993
Earlier work this paper cites.
Motion sketch: Acquisition of visual motion guided behaviors
T. Nakamura and M. Asada · 1995
Earlier work this paper cites.
Transformation Invariance in Pattern Recognition - Tangent distance and Tangent propagation
P. Simard, Y. LeCun, J. Denker, and B. Victorri · 1998
Earlier work this paper cites.
Slow feature analysis: unsupervised learning of invariances
L. Wiskott and T. J. Sejnowski · 2002
Earlier work this paper cites.
Best practices for convolutional neural networks applied to visual document analysis
P. Y. Simard, D. Steinkraus, and J. C. Platt · 2003
Earlier work this paper cites.
Learning methods for generic object recognition with invariance to pose and lighting
Y. LeCun, F. J. Huang, and L. Bottou · 2004
Earlier work this paper cites.
Dimensionality Reduction by Learning an Invariant Mapping
R. Hadsell, S. Chopra, and Y. LeCun · 2006
Earlier work this paper cites.
Local invariant feature detectors: a survey
T. Tuytelaars and K. Mikolajczyk · 2008
Earlier work this paper cites.
Extracting and composing robust features with denoising autoencoders
P. Vincent, H. Larochelle, Y. Bengio, and P.-A. Manzagol · 2008
Earlier work this paper cites.
Deep Learning from Temporal Coherence in Video
H. Mobahi, R. Collobert, and J. Weston · 2009
Earlier work this paper cites.
Understanding the difficulty of training deep feedforward neural networks
X. Glorot and Y. Bengio · 2010
Earlier work this paper cites.

Similar

Figure-Ground Segmentation Improves Handled Object Recognition in Egocentric Video
X. Ren and C. Gu · 2010
Cited alongside, same era.
Sun database: Large-scale scene recognition from abbey to zoo
J. Xiao, J. Hays, K. A. Ehinger, A. Oliva, and A. Torralba · 2010
Cited alongside, same era.
Transforming Auto-Encoders
G. E. Hinton, A. Krizhevsky, and S. D. Wang · 2011
Cited alongside, same era.
Transformation equivariant boltzmann machines
J. J. Kivinen and C. K. Williams · 2011
Cited alongside, same era.
Learning intermediate-level representations of form and motion from natural movies
C. F. Cadieu and B. A. Olshausen · 2012
Cited alongside, same era.
Are we ready for autonomous driving? the KITTI vision benchmark suite
A. Geiger, P. Lenz, and R. Urtasun · 2012
Cited alongside, same era.
Learning rotation-aware features: From invariant priors to equivariant descriptors
U. Schmidt and S. Roth · 2012
Cited alongside, same era.
Learning invariant representations with local transformations
K. Sohn and H. Lee · 2012
Cited alongside, same era.
Moving object segmentation using motor signals
C. Xu, J. Liu, and B. Kuipers · 2012
Cited alongside, same era.
Attention prediction in egocentric video using motion and visual saliency
K. Yamada, Y. Sugano, T. Okabe, Y. Sato, A. Sugimoto, and K. Hiraki · 2012
Cited alongside, same era.
Deep learning of invariant features via simulated fixations in video
Original
W. Zou, S. Zhu, K. Yu, and A. Y. Ng · 2012
Cited alongside, same era.
Watching unlabeled videos helps learn new human actions from very few labeled snapshots
C. Chen and K. Grauman · 2013
Cited alongside, same era.

Then

Vision meets Robotics: The KITTI Dataset
A. Geiger, P. Lenz, C. Stiller, and R. Urtasun · 2013
Later among the works it cites.
Learning to predict gaze in egocentric video
Y. Li, A. Fathi, and J. M. Rehg · 2013
Later among the works it cites.
Learning to relate images
R. Memisevic · 2013
Later among the works it cites.
Discriminative Unsupervised Feature Learning with Convolutional Neural Networks
A. Dosovitskiy, J. T. Springenberg, M. Riedmiller, and T. Brox · 2014
Later among the works it cites.
Unsupervised Learning of Spatiotemporally Coherent Metrics
R. Goroshin, J. Bruna, J. Tompson, D. Eigen, and Y. LeCun · 2014
Later among the works it cites.
Caffe: Convolutional architecture for fast feature embedding
Y. Jia, E. Shelhamer, J. Donahue, S. Karayev, J. Long, R. Girshick, S. Guadarrama, and T. Darrell · 2014
Later among the works it cites.
Slowness and sparseness have diverging effects on complex cell learning
J.-P. Lies, R. M. Häfner, and M. Bethge · 2014
Later among the works it cites.
Modeling Deep Temporal Dependencies with Recurrent Grammar Cells””
V. Michalski, R. Memisevic, and K. Konda · 2014
Later among the works it cites.
Video (language) modeling: a baseline for generative models of natural videos
M. Ranzato, A. Szlam, J. Bruna, M. Mathieu, R. Collobert, and S. Chopra · 2014
Later among the works it cites.
Deep convolutional inverse graphics network
T. D. Kulkarni, W. Whitney, P. Kohli, and J. B. Tenenbaum · 2015
Closest in time.
Understanding image representations by measuring their equivariance and equivalence
K. Lenc and A. Vedaldi · 2015
Closest in time.
3d shapenets for 2.5 d object recognition and next-best-view prediction
Z. Wu, S. Song, A. Khosla, X. Tang, and J. Xiao · 2015
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…