LLaVA-Octopus: Unlocking Instruction-Driven Adaptive Projector Fusion for Video Understanding

Understand

In this paper, we introduce LLaVA-Octopus, a novel video multimodal large language model.

• LLaVA-Octopus adaptively weights features from different visual projectors based on user instructions, enabling us to leverage the complementary strengths of each projector.
• We observe that different visual projectors exhibit distinct characteristics when handling specific tasks.
• For instance, some projectors excel at capturing static details, while others are more effective at processing temporal information, and some are better suited for tasks requiring temporal coherence.