2015

Learning to See by Moving

Agrawal, Pulkit, Carreira, Joao, Malik, Jitendra

Understand

The dominant paradigm for feature learning in computer vision relies on training neural networks for the task of object recognition using millions of hand labelled images.

Is it possible to learn useful features for a diverse set of visual tasks using any other form of supervision? In biology, living organisms developed the ability of visual perception for the purpose of moving and acting in the world.
Drawing inspiration from this observation, in this work we investigate if the awareness of egomotion can be used as a supervisory signal for feature learning.
As opposed to the knowledge of class labels, information about egomotion is freely available to mobile agents.

Built on

The Ecological Approach to Visual Perception
J. J. Gibson · 1979
Earlier work this paper cites.
Active perception
R. Bajcsy · 1988
Earlier work this paper cites.
Auto-association by multilayer perceptrons and singular value decomposition
H. Bourlard and Y. Kamp · 1988
Earlier work this paper cites.
Unsupervised learning
H. Barlow · 1989
Earlier work this paper cites.
Backpropagation applied to handwritten zip code recognition
Y. LeCun, B. Boser, J. S. Denker, D. Henderson, R. E. Howard, W. Hubbard, and L. D. Jackel · 1989
Earlier work this paper cites.
Alvinn: An autonomous land vehicle in a neural network
D. A. Pomerleau · 1989
Earlier work this paper cites.
Emergence of simple-cell receptive field properties by learning a sparse code for natural images
B. A. Olshausen et al · 1996
Earlier work this paper cites.
Gradient-based learning applied to document recognition
Y. LeCun, L. Bottou, Y. Bengio, and P. Haffner · 1998
Earlier work this paper cites.
Object recognition from local scale-invariant features
D. G. Lowe · 1999
Earlier work this paper cites.
Slow feature analysis: Unsupervised learning of invariances
L. Wiskott and T. J. Sejnowski · 2002
Earlier work this paper cites.
Learning a similarity metric discriminatively, with application to face verification
S. Chopra, R. Hadsell, and Y. LeCun · 2005
Earlier work this paper cites.
Beyond bags of features: Spatial pyramid matching for recognizing natural scene categories
S. Lazebnik, C. Schmid, and J. Ponce · 2006
Earlier work this paper cites.
Building the gist of a scene: The role of global image features in recognition
A. Oliva and A. Torralba · 2006
Earlier work this paper cites.

Similar

Online learning for offroad robots: Using spatial label propagation to learn long-range traversability
R. Hadsell, P. Sermanet, J. Ben, A. Erkan, J. Han, B. Flepp, U. Muller, and Y. LeCun · 2007
Cited alongside, same era.
Unsupervised learning of invariant feature hierarchies with applications to object recognition
M. Ranzato, F. J. Huang, Y.-L. Boureau, and Y. LeCun · 2007
Cited alongside, same era.
Convolutional deep belief networks for scalable unsupervised learning of hierarchical representations
H. Lee, R. Grosse, R. Ranganath, and A. Y. Ng · 2009
Cited alongside, same era.
Deep learning from temporal coherence in video
H. Mobahi, R. Collobert, and J. Weston · 2009
Cited alongside, same era.
Deep boltzmann machines
R. Salakhutdinov and G. E. Hinton · 2009
Cited alongside, same era.
Detecting people using mutually consistent poselet activations
L. Bourdev, S. Maji, T. Brox, and J. Malik · 2010
Cited alongside, same era.
Learning to represent spatial transformations with factored higher-order boltzmann machines
R. Memisevic and G. E. Hinton · 2010
Cited alongside, same era.
Vlfeat: An open and portable library of computer vision algorithms
A. Vedaldi and B. Fulkerson · 2010
Cited alongside, same era.
Sun database: Large-scale scene recognition from abbey to zoo
Original
J. Xiao, J. Hays, K. A. Ehinger, A. Oliva, and A. Torralba · 2010
Cited alongside, same era.
City-scale landmark identification on mobile devices
D. M. Chen, G. Baatz, K. Koser, S. S. Tsai, R. Vedantham, T. Pylvanainen, K. Roimela, X. Chen, J. Bach, M. Pollefeys, et al · 2011
Cited alongside, same era.
Transforming auto-encoders
G. E. Hinton, A. Krizhevsky, and S. D. Wang · 2011
Cited alongside, same era.
Unsupervised feature learning and deep learning: A review and new perspectives
Original
Y. Bengio, A. C. Courville, and P. Vincent · 2012
Cited alongside, same era.

Then

Imagenet classification with deep convolutional neural networks
A. Krizhevsky, I. Sutskever, and G. E. Hinton · 2012
Later among the works it cites.
Decaf: A deep convolutional activation feature for generic visual recognition
Original
J. Donahue, Y. Jia, O. Vinyals, J. Hoffman, N. Zhang, E. Tzeng, and T. Darrell · 2013
Later among the works it cites.
Vision meets robotics: The kitti dataset
A. Geiger, P. Lenz, C. Stiller, and R. Urtasun · 2013
Later among the works it cites.
Analyzing the performance of multilayer neural networks for object recognition
P. Agrawal, R. Girshick, and J. Malik · 2014
Later among the works it cites.
Descriptor matching with convolutional neural networks: a comparison to sift
Original
P. Fischer, A. Dosovitskiy, and T. Brox · 2014
Later among the works it cites.
Caffe: Convolutional architecture for fast feature embedding
Y. Jia, E. Shelhamer, J. Donahue, S. Karayev, J. Long, R. Girshick, S. Guadarrama, and T. Darrell · 2014
Later among the works it cites.
Video (language) modeling: a baseline for generative models of natural videos
Original
M. Ranzato, A. Szlam, J. Bruna, M. Mathieu, R. Collobert, and S. Chopra · 2014
Later among the works it cites.
Two-stream convolutional networks for action recognition in videos
K. Simonyan and A. Zisserman · 2014
Later among the works it cites.
Visual scene representations: Sufficiency, minimality, invariance and approximations
Original
S. Soatto · 2014
Later among the works it cites.
Reconstructing pascal voc
S. Vicente, J. Carreira, L. Agapito, and J. Batista · 2014
Later among the works it cites.
Unsupervised feature learning from temporal data
Original
R. Goroshin, J. Bruna, J. Tompson, D. Eigen, and Y. LeCun · 2015
Closest in time.
ImageNet Large Scale Visual Recognition Challenge
O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Z. Huang, A. Karpathy, A. Khosla, M. Bernstein, A. C. Berg, and L. Fei-Fei · 2015
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…