Vision Transformers with Self-Distilled Registers

Understand

Vision Transformers (ViTs) have emerged as the dominant architecture for visual processing tasks, demonstrating excellent scalability with increased training data and model size.

• However, recent work has identified the emergence of artifact tokens in ViTs that are incongruous with local semantics.
• These anomalous tokens degrade ViT performance in tasks that require fine-grained localization or structural coherence.
• An effective mitigation of this issue is the addition of register tokens to ViTs, which implicitly "absorb" the artifact term during training.