2017

Multi-task Self-Supervised Visual Learning

Doersch, Carl, Zisserman, Andrew

Understand

We investigate methods for combining multiple self-supervised tasks--i.e., supervised tasks where data can be collected without manual labeling--in order to train a single visual representation.

First, we provide an apples-to-apples comparison of four different self-supervised tasks using the very deep ResNet-101 architecture.
We then combine tasks to jointly train a network.
We also explore lasso regularization to encourage the network to factorize the information in its representation, and methods for "harmonizing" network inputs in order to learn a more unified representation.

Built on

Learning invariance from transformation sequences
P. Földiák · 1991
Earlier work this paper cites.
Slow feature analysis: Unsupervised learning of invariances
L. Wiskott and T. J. Sejnowski · 2002
Earlier work this paper cites.
Deep learning from temporal coherence in video
H. Mobahi, R. Collobert, and J. Weston · 2009
Earlier work this paper cites.
Hogwild: A lock-free approach to parallelizing stochastic gradient descent
B. Recht, C. Re, S. Wright, and F. Niu · 2011
Earlier work this paper cites.
Large scale distributed deep networks
J. Dean, G. Corrado, R. Monga, K. Chen, M. Devin, M. Mao, A. Senior, P. Tucker, K. Yang, Q. V. Le, et al · 2012
Earlier work this paper cites.
Deep learning of invariant features via simulated fixations in video
W. Y. Zou, A. Y. Ng, S. Zhu, and K. Yu · 2012
Earlier work this paper cites.
Action recognition with improved trajectories
H. Wang and C. Schmid · 2013
Earlier work this paper cites.
Discriminative unsupervised feature learning with convolutional neural networks
A. Dosovitskiy, J. T. Springenberg, M. Riedmiller, and T. Brox · 2014
Earlier work this paper cites.
Overfeat: Integrated recognition, localization and detection using convolutional networks
P. Sermanet, D. Eigen, X. Zhang, M. Mathieu, R. Fergus, and Y. LeCun · 2014
Earlier work this paper cites.
Learning to see by moving
P. Agrawal, J. Carreira, and J. Malik · 2015
Earlier work this paper cites.
Unsupervised visual representation learning by context prediction
C. Doersch, A. Gupta, and A. A. Efros · 2015
Earlier work this paper cites.
Predicting depth, surface normals and semantic labels with a common multi-scale convolutional architecture
D. Eigen and R. Fergus · 2015
Earlier work this paper cites.
Contextual action recognition with R*CNN
G. Gkioxari, R. Girshick, and J. Malik · 2015
Earlier work this paper cites.
Hypercolumns for object segmentation and fine-grained localization
B. Hariharan, P. Arbeláez, R. Girshick, and J. Malik · 2015
Earlier work this paper cites.
Learning image representations tied to ego-motion
D. Jayaraman and K. Grauman · 2015
Earlier work this paper cites.
Faster R-CNN: Towards real-time object detection with region proposal networks
S. Ren, K. He, R. Girshick, and J. Sun · 2015
Earlier work this paper cites.

Similar

Imagenet large scale visual recognition challenge
O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Z. Huang, A. Karpathy, A. Khosla, M. Bernstein, et al · 2015
Cited alongside, same era.
Dense optical flow prediction from a static image
J. Walker, A. Gupta, and M. Hebert · 2015
Cited alongside, same era.
Unsupervised learning of visual representations using videos
X. Wang and A. Gupta · 2015
Cited alongside, same era.
Learning to poke by poking: Experiential learning of intuitive physics
Original
P. Agrawal, A. Nair, P. Abbeel, J. Malik, and S. Levine · 2016
Cited alongside, same era.
Soundnet: Learning sound representations from unlabeled video
Y. Aytar, C. Vondrick, and A. Torralba · 2016
Cited alongside, same era.
Revisiting distributed synchronous SGD
J. Chen, R. Monga, S. Bengio, and R. Jozefowicz · 2016
Cited alongside, same era.
Semi-supervised learning with context-conditional generative adversarial networks
Original
E. Denton, S. Gross, and R. Fergus · 2016
Cited alongside, same era.
Self-supervised video representation learning with odd-one-out networks
Original
B. Fernando, H. Bilen, E. Gavves, and S. Gould · 2016
Cited alongside, same era.
Tensorflow-slim
S. Guadarrama and N. Silberman · 2016
Cited alongside, same era.
Identity mappings in deep residual networks
K. He, X. Zhang, S. Ren, and J. Sun · 2016
Cited alongside, same era.
Ubernet: Training a ‘universal’ convolutional neural network for low-, mid-, and high-level vision using diverse datasets and limited memory
Original
I. Kokkinos · 2016
Cited alongside, same era.
Deeper depth prediction with fully convolutional residual networks
I. Laina, C. Rupprecht, V. Belagiannis, F. Tombari, and N. Navab · 2016
Cited alongside, same era.

Then

Unsupervised learning of visual representations by solving jigsaw puzzles
M. Noroozi and P. Favaro · 2016
Later among the works it cites.
Ambient sound provides supervision for visual learning
A. Owens, J. Wu, J. H. McDermott, W. T. Freeman, and A. Torralba · 2016
Later among the works it cites.
Learning features by watching objects move
Original
D. Pathak, R. Girshick, P. Dollár, T. Darrell, and B. Hariharan · 2016
Later among the works it cites.
Context encoders: Feature learning by inpainting
D. Pathak, P. Krahenbuhl, J. Donahue, T. Darrell, and A. A. Efros · 2016
Later among the works it cites.
Supervision via competition: Robot adversaries for learning tasks
Original
L. Pinto, J. Davidson, and A. Gupta · 2016
Later among the works it cites.
The curious robot: Learning visual representations via physical interactions
L. Pinto, D. Gandhi, Y. Han, Y.-L. Park, and A. Gupta · 2016
Later among the works it cites.
Supersizing self-supervision: Learning to grasp from 50k tries and 700 robot hours
L. Pinto and A. Gupta · 2016
Later among the works it cites.
Progressive neural networks
Original
A. A. Rusu, N. C. Rabinowitz, G. Desjardins, H. Soyer, J. Kirkpatrick, K. Kavukcuoglu, R. Pascanu, and R. Hadsell · 2016
Later among the works it cites.
Inception-v4, inception-resnet and the impact of residual connections on learning
Original
C. Szegedy, S. Ioffe, V. Vanhoucke, and A. Alemi · 2016
Later among the works it cites.
An uncertain future: Forecasting from static images using variational autoencoders
J. Walker, C. Doersch, A. Gupta, and M. Hebert · 2016
Later among the works it cites.
Generic 3D representation via pose estimation and matching
A. R. Zamir, T. Wekel, P. Agrawal, C. Wei, J. Malik, and S. Savarese · 2016
Later among the works it cites.
Understanding deep learning requires rethinking generalization
Original
C. Zhang, S. Bengio, M. Hardt, B. Recht, and O. Vinyals · 2016
Later among the works it cites.
Colorful image colorization
R. Zhang, P. Isola, and A. A. Efros · 2016
Later among the works it cites.
Densely connected convolutional networks
G. Huang, Z. Liu, K. Q. Weinberger, and L. van der Maaten · 2017
Closest in time.
Learning to push by grasping: Using multiple tasks for effective learning
L. Pinto and A. Gupta · 2017
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…