ELSA: Enhanced Local Self-Attention for Vision Transformer

Understand

Self-attention is powerful in modeling long-range dependencies, but it is weak in local finer-level feature learning.

• The performance of local self-attention (LSA) is just on par with convolution and inferior to dynamic filters, which puzzles researchers on whether to use LSA or its counterparts, which one is better, and what makes LSA mediocre.
• To clarify these, we comprehensively investigate LSA and its counterparts from two sides: \emph{channel setting} and \emph{spatial processing}.
• We find that the devil lies in the generation and application of spatial attention, where relative position embeddings and the neighboring filter application are key factors.