Prune Spatio-temporal Tokens by Semantic-aware Temporal Accumulation

Understand

Transformers have become the primary backbone of the computer vision community due to their impressive performance.

• However, the unfriendly computation cost impedes their potential in the video recognition domain.
• To optimize the speed-accuracy trade-off, we propose Semantic-aware Temporal Accumulation score (STA) to prune spatio-temporal tokens integrally.
• STA score considers two critical factors: temporal redundancy and semantic importance.