MMR: Evaluating Reading Ability of Large Multimodal Models

Understand

Large multimodal models (LMMs) have demonstrated impressive capabilities in understanding various types of image, including text-rich images.

• Most existing text-rich image benchmarks are simple extraction-based question answering, and many LMMs now easily achieve high scores.
• This means that current benchmarks fail to accurately reflect performance of different models, and a natural idea is to build a new benchmark to evaluate their complex reasoning and spatial understanding abilities.
• In this work, we propose the Multi-Modal Reading (MMR) benchmark in 11 diverse tasks to evaluate LMMs for text-rich image understanding.