ChartMuseum: Testing Visual Reasoning Capabilities of Large Vision-Language Models

Understand

Chart understanding presents a unique challenge for large vision-language models (LVLMs), as it requires the integration of sophisticated textual and visual reasoning capabilities.

• However, current LVLMs exhibit a notable imbalance between these skills, falling short on visual reasoning that is difficult to perform in text.
• We conduct a case study using a synthetic dataset solvable only through visual reasoning and show that model performance degrades significantly with increasing visual complexity, while human performance remains robust.
• We then introduce ChartMuseum, a new Chart Question Answering (QA) benchmark containing 1,162 expert-annotated questions spanning multiple reasoning types, curated from real-world charts across 184 sources, specifically built to evaluate complex visual and textual reasoning.