Fetching the paper…

Reading the bibliography…

2025

MME-Reasoning: A Comprehensive Benchmark for Logical Reasoning in MLLMs

Yuan, Jiakang, Peng, Tianshuo, Jiang, Yilei et al.

Original
PDF
Post on X

Understand

Logical reasoning is a fundamental aspect of human intelligence and an essential capability for multimodal large language models (MLLMs).

Despite the significant advancement in multimodal reasoning, existing benchmarks fail to comprehensively evaluate their reasoning abilities due to the lack of explicit categorization for logical reasoning types and an unclear understanding of reasoning.
To address these issues, we introduce MME-Reasoning, a comprehensive benchmark designed to evaluate the reasoning ability of MLLMs, which covers all three types of reasoning (i.e., inductive, deductive, and abductive) in its questions.
We carefully curate the data to ensure that each question effectively evaluates reasoning ability rather than perceptual skills or knowledge breadth, and extend the evaluation protocols to cover the evaluation of diverse questions.

MME-Reasoning: A Comprehensive Benchmark for Logical Reasoning in MLLMs · Around

Built on

Illustrations of the Logic of Science
Charles Sanders Peirce · 2014
Earlier work this paper cites.
Clevr: A diagnostic dataset for compositional language and elementary visual reasoning, 2016
Original
Justin Johnson, Bharath Hariharan, Laurens van der Maaten, Li Fei-Fei, C. Lawrence Zitnick, and Ross Girshick · 2016
Earlier work this paper cites.
Scannet: Richly-annotated 3d reconstructions of indoor scenes
Angela Dai, Angel X Chang, Manolis Savva, Maciej Halber, Thomas Funkhouser, and Matthias Nießner · 2017
Earlier work this paper cites.
Towards automatic learning of procedures from web instructional videos
Luowei Zhou, Chenliang Xu, and Jason Corso · 2018
Earlier work this paper cites.
GQA: A new dataset for real-world visual reasoning and compositional question answering
Drew A. Hudson and Christopher D. Manning · 2019
Earlier work this paper cites.
Arkitscenes: A diverse real-world dataset for 3d indoor scene understanding using mobile rgb-d data
Original
Gilad Baruch, Zhuoyuan Chen, Afshin Dehghan, Tal Dimry, Yuri Feigin, Peter Fu, Thomas Gebauer, Brandon Joffe, Daniel Kurz, Arik Schwartz, et al · 2021
Earlier work this paper cites.
Claude, 2022
https://www.anthropic.com/index/introducing-claude Anthropic · 2022
Earlier work this paper cites.
Chain-of-thought prompting elicits reasoning in large language models
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al · 2022
Earlier work this paper cites.
See, think, confirm: Interactive prompting between vision and language models for knowledge-based visual reasoning
Original
Zhenfang Chen, Qinhong Zhou, Yikang Shen, Yining Hong, Hao Zhang, and Chuang Gan · 2023
Earlier work this paper cites.
Mme: A comprehensive evaluation benchmark for multimodal large language models
Original
Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Jinrui Yang, Xiawu Zheng, Ke Li, Xing Sun, et al · 2023
Earlier work this paper cites.
Gemini: a family of highly capable multimodal models
Original
Gemini, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al · 2023
Earlier work this paper cites.
Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts
Original
Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hannaneh Hajishirzi, Hao Cheng, Kai-Wei Chang, Michel Galley, and Jianfeng Gao · 2023
Earlier work this paper cites.
Gpqa: A graduate-level google-proof qa benchmark, 2023
Original
David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R. Bowman · 2023
Earlier work this paper cites.
Thinking like an expert: Multimodal hypergraph-of-thought (hot) reasoning to boost foundation modals
Original
Fanglong Yao, Changyuan Tian, Jintao Liu, Zequn Zhang, Qing Liu, Li Jin, Shuchao Li, Xiaoyu Li, and Xian Sun · 2023
Earlier work this paper cites.
Multimodal chain-of-thought reasoning in language models
Original
Zhuosheng Zhang, Aston Zhang, Mu Li, Hai Zhao, George Karypis, and Alex Smola · 2023
Earlier work this paper cites.
Puzzlevqa: Diagnosing multimodal reasoning challenges of language models with abstract visual patterns
Original
Yew Ken Chia, Vernon Toh Yan Han, Deepanway Ghosal, Lidong Bing, and Soujanya Poria · 2024
Earlier work this paper cites.
Molmo and pixmo: Open weights and open data for state-of-the-art multimodal models
Original
Matt Deitke, Christopher Clark, Sangho Lee, Rohun Tripathi, Yue Yang, Jae Sung Park, Mohammadreza Salehi, Niklas Muennighoff, Kyle Lo, Luca Soldaini, et al · 2024
Earlier work this paper cites.
The llama 3 herd of models
Original
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al · 2024
Earlier work this paper cites.

Similar

Video-mme: The first-ever comprehensive evaluation benchmark of multi-modal llms in video analysis
Original
Chaoyou Fu, Yuhan Dai, Yongdong Luo, Lei Li, Shuhuai Ren, Renrui Zhang, Zihan Wang, Chenyu Zhou, Yunhang Shen, Mengdan Zhang, et al · 2024
Cited alongside, same era.
Av-odyssey bench: Can your multimodal llms really understand audio-visual information?
Original
Kaixiong Gong, Kaituo Feng, Bohao Li, Yibing Wang, Mofan Cheng, Shijia Yang, Jiaming Han, Benyou Wang, Yutong Bai, Zhuoran Yang, et al · 2024
Cited alongside, same era.
Olympiadbench: A challenging benchmark for promoting agi with olympiad-level bilingual multimodal scientific problems
Original
Chaoqun He, Renjie Luo, Yuzhuo Bai, Shengding Hu, Zhen Leng Thai, Junhao Shen, Jinyi Hu, Xu Han, Yujie Huang, Yuxiang Zhang, et al · 2024
Cited alongside, same era.
Visual sketchpad: Sketching as a visual chain of thought for multimodal language models
Original
Yushi Hu, Weijia Shi, Xingyu Fu, Dan Roth, Mari Ostendorf, Luke Zettlemoyer, Noah A Smith, and Ranjay Krishna · 2024

Then

Openvlthinker: An early exploration to complex vision-language reasoning via iterative self-improvement
Original
Yihe Deng, Hritik Bansal, Fan Yin, Nanyun Peng, Wei Wang, and Kai-Wei Chang · 2025
Closest in time.
Virgo: A preliminary exploration on reproducing o1-like mllm
Original
Yifan Du, Zikang Liu, Yifan Li, Wayne Xin Zhao, Yuqi Huo, Bingning Wang, Weipeng Chen, Zheng Liu, Zhongyuan Wang, and Ji-Rong Wen · 2025
Closest in time.
Video-r1: Reinforcing video reasoning in mllms
Original
Kaituo Feng, Kaixiong Gong, Bohao Li, Zonghao Guo, Yibing Wang, Tianshuo Peng, Benyou Wang, and Xiangyu Yue · 2025
Closest in time.
Can mllms reason in multimodality? emma: An enhanced multimodal reasoning benchmark
Original
Yunzhuo Hao, Jiawei Gu, Huichen Will Wang, Linjie Li, Zhengyuan Yang, Lijuan Wang, and Yu Cheng · 2025
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Cited alongside, same era.

Openai o1 system card

Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, et al · 2024

Cited alongside, same era.

Rapguard: Safeguarding multimodal large language models via rationale-aware defensive prompting, 2024

Yilei Jiang, Yingshui Tan, and Xiangyu Yue · 2024

Cited alongside, same era.

Llava-onevision: Easy visual task transfer

Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Yanwei Li, Ziwei Liu, and Chunyuan Li · 2024

Cited alongside, same era.

Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts

Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hannaneh Hajishirzi, Hao Cheng, Kai-Wei Chang, Michel Galley, and Jianfeng Gao · 2024

Cited alongside, same era.

Hello gpt4-o. https://openai.com/index/hello-gpt-4o/, 2024

OpenAI · 2024

Cited alongside, same era.

Chimera: Improving generalist model with domain-specific experts

Tianshuo Peng, Mingsheng Li, Hongbin Zhou, Renqiu Xia, Renrui Zhang, Lei Bai, Song Mao, Bin Wang, Conghui He, Aojun Zhou, et al · 2024

Cited alongside, same era.

Qvq: To see the world with wisdom, December 2024

Qwen Team · 2024

Cited alongside, same era.

Mulberry: Empowering mllm with o1-like reasoning and reflection via collective monte carlo tree search

Original

Huanjin Yao, Jiaxing Huang, Wenhao Wu, Jingyi Zhang, Yibo Wang, Shunyu Liu, Yingjie Wang, Yuxin Song, Haocheng Feng, Li Shen, et al · 2024

Cited alongside, same era.

Vision-r1: Incentivizing reasoning capability in multimodal large language models

Wenxuan Huang, Bohan Jia, Zijie Zhai, Shaosheng Cao, Zheyu Ye, Fei Zhao, Zhe Xu, Yao Hu, and Shaohui Lin · 2025

Closest in time.

Mme-cot: Benchmarking chain-of-thought in large multimodal models for reasoning quality, robustness, and efficiency, 2025

Dongzhi Jiang, Renrui Zhang, Ziyu Guo, Yanwei Li, Yu Qi, Xinyan Chen, Liuhui Wang, Jianhan Jin, Claire Guo, Shen Yan, Bo Zhang, Chaoyou Fu, Peng Gao, and Hongsheng Li · 2025

Closest in time.

Omnicaptioner: One captioner to rule them all

Yiting Lu, Jiakang Yuan, Zhen Li, Shitian Zhao, Qi Qin, Xinyue Li, Le Zhuo, Licheng Wen, Dongyang Liu, Yuewen Cao, et al · 2025

Closest in time.

Mm-eureka: Exploring visual aha moment with rule-based large-scale reinforcement learning, 2025

Fanqing Meng, Lingxiao Du, Zongkai Liu, Zhixiang Zhou, Quanfeng Lu, Daocheng Fu, Botian Shi, Wenhai Wang, Junjun He, Kaipeng Zhang, Ping Luo, Yu Qiao, Qiaosheng Zhang, and Wenqi Shao · 2025

Closest in time.

Vlm-r1: A stable and generalizable r1-style large vision-language model, 2025

Haozhan Shen, Peng Liu, Jingcheng Li, Chunxin Fang, Yibo Ma, Jiajia Liao, Qiaoli Shen, Zilun Zhang, Kangjia Zhao, Qianqian Zhang, Ruochen Xu, and Tiancheng Zhao · 2025

Closest in time.

Visualpuzzles: Decoupling multimodal reasoning evaluation from domain knowledge

Yueqi Song, Tianyue Ou, Yibo Kong, Zecheng Li, Graham Neubig, and Xiang Yue · 2025

Closest in time.

Llamav-o1: Rethinking step-by-step visual reasoning in llms

Omkar Thawakar, Dinura Dissanayake, Ketan More, Ritesh Thawkar, Ahmed Heakl, Noor Ahsan, Yuhao Li, Mohammed Zumri, Jean Lahoud, Rao Muhammad Anwer, et al · 2025

Closest in time.

Vl-rethinker: Incentivizing self-reflection of vision-language models with reinforcement learning

Haozhe Wang, Chao Qu, Zuming Huang, Wei Chu, Fangzhen Lin, and Wenhu Chen · 2025

Closest in time.

R1-onevision: Advancing generalized multimodal reasoning through cross-modal formalization

Yi Yang, Xiaoxuan He, Hongkun Pan, Xiyan Jiang, Yan Deng, Xingtao Yang, Haoyu Lu, Dacheng Yin, Fengyun Rao, Minfeng Zhu, et al · 2025

Closest in time.

Lmm-r1: Empowering 3b lmms with strong reasoning abilities through two-stage rule-based rl, 2025

Peng Yingzhe, Zhang Gongrui, Zhang Miaosen, You Zhiyuan, Liu Jie, Zhu Qipeng, Yang Kai, Xu Xingzhong, Geng Xin, and Yang Xu · 2025

Closest in time.

Seg-zero: Reasoning-chain guided segmentation via cognitive reinforcement, 2025

Liu Yuqi, Peng Bohao, Zhong Zhisheng, Yue Zihao, Lu Fanbin, Yu Bei, and Jia Jiaya · 2025

Closest in time.

R1-omni: Explainable omni-multimodal emotion recognition with reinforcement learning, 2025

Jiaxing Zhao, Xihan Wei, and Liefeng Bo · 2025

Closest in time.

R1-zero’s "aha moment" in visual reasoning on a 2b non-sft model, 2025

Hengguang Zhou, Xirui Li, Ruochen Wang, Minhao Cheng, Tianyi Zhou, and Cho-Jui Hsieh · 2025

Closest in time.

Internvl3: Exploring advanced training and test-time recipes for open-source multimodal models

Jinguo Zhu, Weiyun Wang, Zhe Chen, Zhaoyang Liu, Shenglong Ye, Lixin Gu, Yuchen Duan, Hao Tian, Weijie Su, Jie Shao, et al · 2025

Closest in time.