XOV-Action: Towards Generalizable Open-Vocabulary Action Recognition

Understand

Inspired by the impressive success of image-text foundation models, recent works have proposed to adapt these foundation models to video data, leading to efficient and effective video models for open-vocabulary action recognition.

• However, through a comprehensive evaluation, our work finds that state-of-the-art open-vocabulary action recognition models still struggle with generalization to video domains that they have not encountered.
• To address this limitation, we introduce \textit{generalizable open-vocabulary action recognition}, which aims to develop action recognition models capable of generalizing to both novel action categories and unseen video domains.
• Our work contributes a novel model named XOV-Action to overcome two critical challenges: (1) understanding novel action concepts of open-set categories, and (2) mitigating the scenario discrepancy between training and test datasets.