Vision Transformer with Sparse Scan Prior

Understand

In recent years, Transformers have achieved remarkable progress in computer vision tasks.

• However, their global modeling often comes with substantial computational overhead, in stark contrast to the human eye's efficient information processing.
• Inspired by the human eye's sparse scanning mechanism, we propose a \textbf{S}parse \textbf{S}can \textbf{S}elf-\textbf{A}ttention mechanism ($\rm{S}^3\rm{A}$).
• This mechanism predefines a series of Anchors of Interest for each token and employs local attention to efficiently model the spatial information around these anchors, avoiding redundant global modeling and excessive focus on local information.