Extreme Masking for Learning Instance and Distributed Visual Representations

Understand

The paper presents a scalable approach for learning spatially distributed visual representations over individual tokens and a holistic instance representation simultaneously.

• We use self-attention blocks to represent spatially distributed tokens, followed by cross-attention blocks to aggregate the holistic image instance.
• The core of the approach is the use of extremely large token masking (75\%-90\%) as the data augmentation for supervision.
• Our model, named ExtreMA, follows the plain BYOL approach where the instance representation from the unmasked subset is trained to predict that from the intact input.