VisionZip: Longer is Better but Not Necessary in Vision Language Models

Understand

Recent advancements in vision-language models have enhanced performance by increasing the length of visual tokens, making them much longer than text tokens and significantly raising computational costs.

• However, we observe that the visual tokens generated by popular vision encoders, such as CLIP and SigLIP, contain significant redundancy.
• To address this, we introduce VisionZip, a simple yet effective method that selects a set of informative tokens for input to the language model, reducing visual token redundancy and improving efficiency while maintaining model performance.
• The proposed VisionZip can be widely applied to image and video understanding tasks and is well-suited for multi-turn dialogues in real-world scenarios, where previous methods tend to underperform.