KVT: k-NN Attention for Boosting Vision Transformers

Understand

Convolutional Neural Networks (CNNs) have dominated computer vision for years, due to its ability in capturing locality and translation invariance.

• Recently, many vision transformer architectures have been proposed and they show promising performance.
• A key component in vision transformers is the fully-connected self-attention which is more powerful than CNNs in modelling long range dependencies.
• However, since the current dense self-attention uses all image patches (tokens) to compute attention matrix, it may neglect locality of images patches and involve noisy tokens (e.g., clutter background and occlusion), leading to a slow training process and potential degradation of performance.