Percept, Chat, and then Adapt: Multimodal Knowledge Transfer of Foundation Models for Open-World Video Recognition

Understand

Open-world video recognition is challenging since traditional networks are not generalized well on complex environment variations.

• Alternatively, foundation models with rich knowledge have recently shown their generalization power.
• However, how to apply such knowledge has not been fully explored for open-world video recognition.
• To this end, we propose a generic knowledge transfer pipeline, which progressively exploits and integrates external multimodal knowledge from foundation models to boost open-world video recognition.