Video Action Recognition with Attentive Semantic Units

Understand

Visual-Language Models (VLMs) have significantly advanced action video recognition.

• Supervised by the semantics of action labels, recent works adapt the visual branch of VLMs to learn video representations.
• Despite the effectiveness proved by these works, we believe that the potential of VLMs has yet to be fully harnessed.
• In light of this, we exploit the semantic units (SU) hiding behind the action labels and leverage their correlations with fine-grained items in frames for more accurate action recognition.