ReTaKe: Reducing Temporal and Knowledge Redundancy for Long Video Understanding

Understand

Video Large Language Models (VideoLLMs) have made significant strides in video understanding but struggle with long videos due to the limitations of their backbone LLMs.

• Existing solutions rely on length extrapolation, which is memory-constrained, or visual token compression, which primarily leverages low-level temporal redundancy while overlooking the more effective high-level knowledge redundancy.
• To address this, we propose $\textbf{ReTaKe}$, a training-free method with two novel modules DPSelect and PivotKV, to jointly reduce both temporal visual redundancy and knowledge redundancy for video compression.
• To align with the way of human temporal perception, DPSelect identifies keyframes based on inter-frame distance peaks.