Patch-level Representation Learning for Self-supervised Vision Transformers

Understand

Recent self-supervised learning (SSL) methods have shown impressive results in learning visual representations from unlabeled images.

• This paper aims to improve their performance further by utilizing the architectural advantages of the underlying neural network, as the current state-of-the-art visual pretext tasks for SSL do not enjoy the benefit, i.e., they are architecture-agnostic.
• In particular, we focus on Vision Transformers (ViTs), which have gained much attention recently as a better architectural choice, often outperforming convolutional networks for various visual tasks.
• The unique characteristic of ViT is that it takes a sequence of disjoint patches from an image and processes patch-level representations internally.