Are We Using the Right Benchmark: An Evaluation Framework for Visual Token Compression Methods

Understand

Recent efforts to accelerate inference in Multimodal Large Language Models (MLLMs) have largely focused on visual token compression.

• The effectiveness of these methods is commonly evaluated by measuring the accuracy drop on existing MLLM benchmarks before and after compression.
• However, these benchmarks are originally designed to assess general perception and reasoning abilities, rather than the specific challenges posed by visual token compression, leading to a fundamental task mismatch.
• In this work, we uncover a counterintuitive yet consistent phenomenon: simple image downsampling outperforms many advanced visual token compression methods across multiple widely used benchmarks.