2021

Barlow Twins: Self-Supervised Learning via Redundancy Reduction

Zbontar, Jure, Jing, Li, Misra, Ishan et al.

Understand

Self-supervised learning (SSL) is rapidly closing the gap with supervised methods on large computer vision benchmarks.

A successful approach to SSL is to learn embeddings which are invariant to distortions of the input sample.
However, a recurring issue with this approach is the existence of trivial constant solutions.
Most current methods avoid such solutions by careful implementation details.

Built on

Possible Principles Underlying the Transformations of Sensory Messages
Barlow, H. B · 1961
Earlier work this paper cites.
Self-organization in a perceptual network
Linsker, R · 1988
Earlier work this paper cites.
Forming sparse representations by local anti-Hebbian learning
Földiák, P · 1990
Earlier work this paper cites.
Discovering viewpoint-invariant relationships that characterize objects
Zemel, R. and Hinton, G. E · 1990
Earlier work this paper cites.
Self-organizing neural network that discovers surfaces in random-dot stereograms
Becker, S. and Hinton, G. E · 1992
Earlier work this paper cites.
Redundancy Reduction as a Strategy for Unsupervised Learning
Redlich, A. N · 1993
Earlier work this paper cites.
Supervised Factorial Learning
Redlich, A. N · 1993
Earlier work this paper cites.
Semilinear Predictability Minimization Produces Well-Known Feature Detectors
Schmidhuber, J., Eldracher, M., and Foltin, B · 1996
Earlier work this paper cites.
Non-linear Feature Extraction by Redundancy Reduction in an Unsupervised Stochastic Neural Network
Deco, G. and Parra, L · 1997
Earlier work this paper cites.
The information bottleneck method
Tishby, N., Pereira, F. C., and Bialek, W · 2000
Earlier work this paper cites.
Redundancy reduction revisited
Barlow, H · 2001
Earlier work this paper cites.
Natural signal statistics and sensory gain control
Schwartz, O. and Simoncelli, E. P · 2001
Earlier work this paper cites.
A simple framework for contrastive learning of visual representations
Original
Chen, T., Kornblith, S., Norouzi, M., and Hinton, G · 2002
Earlier work this paper cites.
Improved baselines with momentum contrastive learning
Original
Chen, X., Fan, H., Girshick, R., and He, K · 2003
Earlier work this paper cites.
Dimensionality reduction by learning an invariant mapping
Hadsell, R., Chopra, S., and LeCun, Y · 2006
Earlier work this paper cites.
Imagenet: A large-scale hierarchical image database
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., and Fei-Fei, L · 2009
Earlier work this paper cites.
The pascal visual object classes (voc) challenge
Everingham, M., Van Gool, L., Williams, C. K., Winn, J., and Zisserman, A · 2010
Earlier work this paper cites.
Noise-contrastive estimation: A new estimation principle for unnormalized statistical models
Gutmann, M. and Hyvärinen, A · 2010
Earlier work this paper cites.

Similar

Microsoft coco: Common objects in context
Lin, T.-Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., and Zitnick, C. L · 2014
Cited alongside, same era.
Learning deep features for scene recognition using places database
Zhou, B., Lapedriza, A., Xiao, J., Torralba, A., and Oliva, A · 2014
Cited alongside, same era.
Law of Log Determinant of Sample Covariance Matrix and Optimal Estimation of Differential Entropy for High-Dimensional Gaussian Distributions
Cai, T. T., Liang, T., and Zhou, H. H · 2015
Cited alongside, same era.
Faster r-cnn: Towards real-time object detection with region proposal networks
Ren, S., He, K., Girshick, R., and Sun, J · 2015
Cited alongside, same era.
Deep Learning and the Information Bottleneck Principle
Original
Tishby, N. and Zaslavsky, N · 2015
Cited alongside, same era.
Deep residual learning for image recognition
He, K., Zhang, X., Ren, S., and Sun, J · 2016
Cited alongside, same era.
Sgdr: Stochastic gradient descent with warm restarts
Original
Loshchilov, I. and Hutter, F · 2016
Cited alongside, same era.
End-to-end Optimized Image Compression
Ballé, J., Laparra, V., and Simoncelli, E. P · 2017
Cited alongside, same era.
Unsupervised learning by predicting noise
Bojanowski, P. and Joulin, A · 2017
Cited alongside, same era.
Accurate, large minibatch sgd: Training imagenet in 1 hour
Original
Goyal, P., Dollár, P., Girshick, R., Noordhuis, P., Wesolowski, L., Kyrola, A., Tulloch, A., Jia, Y., and He, K · 2017
Cited alongside, same era.
Mask r-cnn
He, K., Gkioxari, G., Dollár, P., and Girshick, R · 2017
Cited alongside, same era.
Large batch training of convolutional networks
Original
You, Y., Gitman, I., and Ginsburg, B · 2017
Cited alongside, same era.

Then

Momentum contrast for unsupervised visual representation learning
Original
He, K., Fan, H., Wu, Y., Xie, S., and Girshick, R · 2019
Later among the works it cites.
Data-efficient image recognition with contrastive predictive coding
Original
Hénaff, O. J., Razavi, A., Doersch, C., Eslami, S., and Oord, A. v. d · 2019
Later among the works it cites.
Self-supervised learning of pretext-invariant representations
Original
Misra, I. and van der Maaten, L · 2019
Later among the works it cites.
Detectron2
Wu, Y., Kirillov, A., Massa, F., Lo, W.-Y., and Girshick, R · 2019
Later among the works it cites.
S4l: Self-supervised semi-supervised learning
Zhai, X., Oliver, A., Kolesnikov, A., and Beyer, L · 2019
Later among the works it cites.
Self-labelling via simultaneous clustering and representation learning
Asano, Y. M., Rupprecht, C., and Vedaldi, A · 2020
Later among the works it cites.
Unsupervised learning of visual features by contrasting cluster assignments
Caron, M., Misra, I., Mairal, J., Goyal, P., Bojanowski, P., and Joulin, A · 2020
Later among the works it cites.
Exploring simple siamese representation learning
Original
Chen, X. and He, K · 2020
Later among the works it cites.
Whitening for Self-Supervised Representation Learning
Original
Ermolov, A., Siarohin, A., Sangineto, E., and Sebe, N · 2020
Later among the works it cites.
Understanding self-supervised and contrastive learning with “Bootstrap Your Own Latent” (BYOL)
Fetterman, A. and Albrecht, J · 2020
Later among the works it cites.
Bootstrap your own latent: A new approach to self-supervised learning
Grill, J.-B., Strub, F., Altché, F., Tallec, C., Richemond, P. H., Buchatskaya, E., Doersch, C., Pires, B. A., Guo, Z. D., Azar, M. G., et al · 2020
Later among the works it cites.
A unified theory of early visual representations from retina to cortex through anatomically constrained deep CNNs
Lindsey, J., Ocko, S. A., Ganguli, S., and Deny, S · 2020
Later among the works it cites.
BYOL works even without batch statistics
Original
Richemond, P. H., Grill, J.-B., Altché, F., Tallec, C., Strub, F., Brock, A., Smith, S., De, S., Pascanu, R., Piot, B., and Valko, M · 2020
Later among the works it cites.
Understanding Self-supervised Learning with Dual Deep Networks
Original
Tian, Y., Yu, L., Chen, X., and Ganguli, S · 2020
Later among the works it cites.
Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hypersphere
Original
Wang, T. and Isola, P · 2020
Later among the works it cites.
Contrastive clustering
Li, Y., Hu, P., Liu, Z., Peng, D., Zhou, J. T., and Peng, X · 2021
Closest in time.
Understanding self-supervised Learning Dynamics without Contrastive Pairs
Original
Tian, Y., Chen, X., and Ganguli, S · 2021
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…