DynamicViT: Efficient Vision Transformers with Dynamic Token Sparsification

Understand

Attention is sparse in vision transformers.

• We observe the final prediction in vision transformers is only based on a subset of most informative tokens, which is sufficient for accurate image recognition.
• Based on this observation, we propose a dynamic token sparsification framework to prune redundant tokens progressively and dynamically based on the input.
• Specifically, we devise a lightweight prediction module to estimate the importance score of each token given the current features.