Masked Siamese ConvNets

Understand

Self-supervised learning has shown superior performances over supervised methods on various vision benchmarks.

• The siamese network, which encourages embeddings to be invariant to distortions, is one of the most successful self-supervised visual representation learning approaches.
• Among all the augmentation methods, masking is the most general and straightforward method that has the potential to be applied to all kinds of input and requires the least amount of domain knowledge.
• However, masked siamese networks require particular inductive bias and practically only work well with Vision Transformers.