LLaVA-MLB: Mitigating and Leveraging Attention Bias for Training-Free Video LLMs

Understand

Training-free video large language models (LLMs) leverage pretrained Image LLMs to process video content without the need for further training.

• A key challenge in such approaches is the difficulty of retaining essential visual and temporal information, constrained by the token limits in Image LLMs.
• To address this, we propose a two-stage method for selecting query-relevant tokens based on the LLM attention scores: compressing the video sequence and then expanding the sequence.
• However, during the compression stage, Image LLMs often exhibit a positional attention bias in video sequences, where attention is overly concentrated on later frames, causing early-frame information to be underutilized.