Video-RAG: Visually-aligned Retrieval-Augmented Long Video Comprehension

Understand

Existing large video-language models (LVLMs) struggle to comprehend long videos correctly due to limited context.

• To address this problem, fine-tuning long-context LVLMs and employing GPT-based agents have emerged as promising solutions.
• However, fine-tuning LVLMs would require extensive high-quality data and substantial GPU resources, while GPT-based agents would rely on proprietary models (e.g., GPT-4o).
• In this paper, we propose Video Retrieval-Augmented Generation (Video-RAG), a training-free and cost-effective pipeline that employs visually-aligned auxiliary texts to help facilitate cross-modality alignment while providing additional information beyond the visual content.