ClusTR: Exploring Efficient Self-attention via Clustering for Vision Transformers

Understand

Although Transformers have successfully transitioned from their language modelling origins to image-based applications, their quadratic computational complexity remains a challenge, particularly for dense prediction.

• In this paper we propose a content-based sparse attention method, as an alternative to dense self-attention, aiming to reduce the computation complexity while retaining the ability to model long-range dependencies.
• Specifically, we cluster and then aggregate key and value tokens, as a content-based method of reducing the total token count.
• The resulting clustered-token sequence retains the semantic diversity of the original signal, but can be processed at a lower computational cost.