2020

How Useful is Self-Supervised Pretraining for Visual Tasks?

Newell, Alejandro, Deng, Jia

Understand

Recent advances have spurred incredible progress in self-supervised pretraining for vision.

We investigate what factors may play a role in the utility of these pretraining methods for practitioners.
To do this, we evaluate various self-supervised algorithms across a comprehensive array of synthetic datasets and downstream tasks.
We prepare a suite of synthetic data that enables an endless supply of annotated images as well as full control over dataset difficulty.

Built on

Reducing the dimensionality of data with neural networks
Geoffrey E Hinton and Ruslan R Salakhutdinov · 2006
Earlier work this paper cites.
Imagenet: A large-scale hierarchical image database
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei · 2009
Earlier work this paper cites.
Decaf: a deep convolutional activation feature for generic visual recognition. corr (2013)
Original
J Donahue, Y Jia, O Vinyals, J Hoffman, N Zhang, E Tzeng, and T Darrell · 2013
Earlier work this paper cites.
Auto-encoding variational bayes
Original
Diederik P Kingma and Max Welling · 2013
Earlier work this paper cites.
Describing textures in the wild
M. Cimpoi, S. Maji, I. Kokkinos, S. Mohamed, , and A. Vedaldi · 2014
Earlier work this paper cites.
Discriminative unsupervised feature learning with convolutional neural networks
Alexey Dosovitskiy, Jost Tobias Springenberg, Martin Riedmiller, and Thomas Brox · 2014
Earlier work this paper cites.
Rich feature hierarchies for accurate object detection and semantic segmentation
Ross Girshick, Jeff Donahue, Trevor Darrell, and Jitendra Malik · 2014
Earlier work this paper cites.
ShapeNet: An Information-Rich 3D Model Repository
Original
Angel X. Chang, Thomas Funkhouser, Leonidas Guibas, Pat Hanrahan, Qixing Huang, Zimo Li, Silvio Savarese, Manolis Savva, Shuran Song, Hao Su, Jianxiong Xiao, Li Yi, and Fisher Yu · 2015
Earlier work this paper cites.
Unsupervised visual representation learning by context prediction
Carl Doersch, Abhinav Gupta, and Alexei A Efros · 2015
Earlier work this paper cites.
Deep residual learning for image recognition. computer vision and pattern recognition (cvpr)
K He, X Zhang, S Ren, and J Sun · 2015
Earlier work this paper cites.
Unsupervised learning of visual representations using videos
Xiaolong Wang and Abhinav Gupta · 2015
Earlier work this paper cites.
Unsupervised learning of visual representations by solving jigsaw puzzles
Mehdi Noroozi and Paolo Favaro · 2016
Earlier work this paper cites.
Context encoders: Feature learning by inpainting
Deepak Pathak, Philipp Krahenbuhl, Jeff Donahue, Trevor Darrell, and Alexei A Efros · 2016
Earlier work this paper cites.
Colorful image colorization
Richard Zhang, Phillip Isola, and Alexei A Efros · 2016
Earlier work this paper cites.
Unsupervised learning by predicting noise
Piotr Bojanowski and Armand Joulin · 2017
Earlier work this paper cites.
Improved regularization of convolutional neural networks with cutout
Original
Terrance DeVries and Graham W Taylor · 2017
Earlier work this paper cites.
Multi-task self-supervised visual learning
Carl Doersch and Andrew Zisserman · 2017
Earlier work this paper cites.

Similar

Revisiting unreasonable effectiveness of data in deep learning era
Chen Sun, Abhinav Shrivastava, Saurabh Singh, and Abhinav Gupta · 2017
Cited alongside, same era.
Split-brain autoencoders: Unsupervised learning by cross-channel prediction
Richard Zhang, Phillip Isola, and Alexei A Efros · 2017
Cited alongside, same era.
Unsupervised learning of depth and ego-motion from video
Tinghui Zhou, Matthew Brown, Noah Snavely, and David G Lowe · 2017
Cited alongside, same era.
Deep clustering for unsupervised learning of visual features
Mathilde Caron, Piotr Bojanowski, Armand Joulin, and Matthijs Douze · 2018
Cited alongside, same era.
Blender - a 3D modelling and rendering package
Blender Online Community · 2018
Cited alongside, same era.
Unsupervised representation learning by predicting image rotations
Original
Spyros Gidaris, Praveer Singh, and Nikos Komodakis · 2018
Cited alongside, same era.
Tune: A research platform for distributed model selection and training
Original
Richard Liaw, Eric Liang, Robert Nishihara, Philipp Moritz, Joseph E Gonzalez, and Ion Stoica · 2018
Cited alongside, same era.
Challenging common assumptions in the unsupervised learning of disentangled representations
Original
Francesco Locatello, Stefan Bauer, Mario Lucic, Sylvain Gelly, Bernhard Schölkopf, and Olivier Bachem · 2018
Cited alongside, same era.
Exploring the limits of weakly supervised pretraining
Dhruv Mahajan, Ross Girshick, Vignesh Ramanathan, Kaiming He, Manohar Paluri, Yixuan Li, Ashwin Bharambe, and Laurens van der Maaten · 2018
Cited alongside, same era.
Realistic evaluation of deep semi-supervised learning algorithms
Avital Oliver, Augustus Odena, Colin A Raffel, Ekin Dogus Cubuk, and Ian Goodfellow · 2018
Cited alongside, same era.
Representation learning with contrastive predictive coding
Original
Aaron van den Oord, Yazhe Li, and Oriol Vinyals · 2018
Cited alongside, same era.
How to train your resnet, 2018
David Page · 2018
Cited alongside, same era.

Then

Learning representations by maximizing mutual information across views
Original
Philip Bachman, R Devon Hjelm, and William Buchwalter · 2019
Later among the works it cites.
Mixmatch: A holistic approach to semi-supervised learning
Original
David Berthelot, Nicholas Carlini, Ian Goodfellow, Nicolas Papernot, Avital Oliver, and Colin Raffel · 2019
Later among the works it cites.
Scaling and benchmarking self-supervised visual representation learning
Original
Priya Goyal, Dhruv Mahajan, Abhinav Gupta, and Ishan Misra · 2019
Later among the works it cites.
Momentum contrast for unsupervised visual representation learning
Original
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick · 2019
Later among the works it cites.
Rethinking imagenet pre-training
Kaiming He, Ross Girshick, and Piotr Dollár · 2019
Later among the works it cites.
Data-efficient image recognition with contrastive predictive coding
Original
Olivier J Hénaff, Ali Razavi, Carl Doersch, SM Eslami, and Aaron van den Oord · 2019
Later among the works it cites.
Self-supervised visual feature learning with deep neural networks: A survey
Original
Longlong Jing and Yingli Tian · 2019
Later among the works it cites.
Large scale learning of general visual representations for transfer
Original
Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Joan Puigcerver, Jessica Yung, Sylvain Gelly, and Neil Houlsby · 2019
Later among the works it cites.
Revisiting self-supervised visual representation learning
Original
Alexander Kolesnikov, Xiaohua Zhai, and Lucas Beyer · 2019
Later among the works it cites.
Probing the state of the art: A critical look at visual representation evaluation
Original
Cinjon Resnick, Zeping Zhan, and Joan Bruna · 2019
Later among the works it cites.
Contrastive multiview coding
Original
Yonglong Tian, Dilip Krishnan, and Phillip Isola · 2019
Later among the works it cites.
Self-training with noisy student improves imagenet classification
Original
Qizhe Xie, Eduard Hovy, Minh-Thang Luong, and Quoc V Le · 2019
Later among the works it cites.
Billion-scale semi-supervised learning for image classification
Original
I Zeki Yalniz, Hervé Jégou, Kan Chen, Manohar Paluri, and Dhruv Mahajan · 2019
Later among the works it cites.
Unsupervised embedding learning via invariant and spreading instance feature
Mang Ye, Xu Zhang, Pong C Yuen, and Shih-Fu Chang · 2019
Later among the works it cites.
The visual task adaptation benchmark
Original
Xiaohua Zhai, Joan Puigcerver, Alexander Kolesnikov, Pierre Ruyssen, Carlos Riquelme, Mario Lucic, Josip Djolonga, Andre Susano Pinto, Maxim Neumann, Alexey Dosovitskiy, et al · 2019
Later among the works it cites.
Local aggregation for unsupervised learning of visual embeddings
Chengxu Zhuang, Alex Lin Zhai, and Daniel Yamins · 2019
Later among the works it cites.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…