CrossLMM: Decoupling Long Video Sequences from LMMs via Dual Cross-Attention Mechanisms

Understand

The advent of Large Multimodal Models (LMMs) has significantly enhanced Large Language Models (LLMs) to process and interpret diverse data modalities (e.g., image and video).

• However, as input complexity increases, particularly with long video sequences, the number of required tokens has grown significantly, leading to quadratically computational costs.
• This has made the efficient compression of video tokens in LMMs, while maintaining performance integrity, a pressing research challenge.
• In this paper, we introduce CrossLMM, decoupling long video sequences from LMMs via a dual cross-attention mechanism, which substantially reduces visual token quantity with minimal performance degradation.