Do You See Me : A Multidimensional Benchmark for Evaluating Visual Perception in Multimodal LLMs

Understand

Multimodal Large Language Models (MLLMs) show reasoning promise, yet their visual perception is a critical bottleneck.

• Strikingly, MLLMs can produce correct answers even while misinterpreting crucial visual elements, masking these underlying failures.
• Our preliminary study on a joint perception-reasoning dataset revealed that for one leading MLLM, 29% of its correct answers to reasoning questions still exhibited visual perception errors.
• To systematically address this, we introduce "Do You See Me", a scalable benchmark with 1,758 images and 2,612 questions.