Understanding The Robustness in Vision Transformers

Understand

Recent studies show that Vision Transformers(ViTs) exhibit strong robustness against various corruptions.

• Although this property is partly attributed to the self-attention mechanism, there is still a lack of systematic understanding.
• In this paper, we examine the role of self-attention in learning robust representations.
• Our study is motivated by the intriguing properties of the emerging visual grouping in Vision Transformers, which indicates that self-attention may promote robustness through improved mid-level representations.