HumanEval-V: Benchmarking High-Level Visual Reasoning with Complex Diagrams in Coding Tasks

On the measure of intelligence

Original

François Chollet. 2019 · 1911

Earlier work this paper cites.

Cyclomatic complexity density and software maintenance productivity

Geoffrey K Gill and Chris F Kemerer. 1991 · 1991

Earlier work this paper cites.

A diagram is worth a dozen images

Aniruddha Kembhavi, Mike Salvato, Eric Kolve, Minjoon Seo, Hannaneh Hajishirzi, and Ali Farhadi. 2016 · 2016

Earlier work this paper cites.

Raven: A dataset for relational and analogical visual reasoning

Chi Zhang, Feng Gao, Baoxiong Jia, Yixin Zhu, and Song-Chun Zhu. 2019 · 2019

Earlier work this paper cites.

Bongard-logo: A new benchmark for human-level concept learning and reasoning

Weili Nie, Zhiding Yu, Lei Mao, Ankit B Patel, Yuke Zhu, and Anima Anandkumar. 2020 · 2020

Earlier work this paper cites.

Evaluating large language models trained on code

Original

Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. 2021 · 2021

Earlier work this paper cites.

Iconqa: A new benchmark for abstract diagram understanding and visual language reasoning

Original

Pan Lu, Liang Qiu, Jiaqi Chen, Tony Xia, Yizhou Zhao, Wei Zhang, Zhou Yu, Xiaodan Liang, and Song-Chun Zhu. 2021 · 2021

Earlier work this paper cites.

Codet: Code generation with generated tests

Original

Bei Chen, Fengji Zhang, Anh Nguyen, Daoguang Zan, Zeqi Lin, Jian-Guang Lou, and Weizhu Chen. 2022 · 2022

Earlier work this paper cites.

Learn to explain: Multimodal reasoning via thought chains for science question answering

Pan Lu, Swaroop Mishra, Tanglin Xia, Liang Qiu, Kai-Wei Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. 2022 · 2022

Earlier work this paper cites.

Chartqa: A benchmark for question answering about charts with visual and logical reasoning

Original

Ahmed Masry, Do Xuan Long, Jia Qing Tan, Shafiq Joty, and Enamul Hoque. 2022 · 2022

Earlier work this paper cites.

Chain-of-thought prompting elicits reasoning in large language models

Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. 2022 · 2022

Earlier work this paper cites.

Hallusionbench: An advanced diagnostic suite for entangled language hallucination and visual illusion in large vision-language models

Original

Tianrui Guan, Fuxiao Liu, Xiyang Wu, Ruiqi Xian, Zongxia Li, Xiaoyu Liu, Xijun Wang, Lichang Chen, Furong Huang, Yaser Yacoob, et al. 2023 · 2023

Earlier work this paper cites.

Seed-bench: Benchmarking multimodal llms with generative comprehension

Original

Bohao Li, Rui Wang, Guangzhi Wang, Yuying Ge, Yixiao Ge, and Ying Shan. 2023 · 2023

Earlier work this paper cites.

Mmbench: Is your multi-modal model an all-around player?

Original

Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, et al. 2023 · 2023

Earlier work this paper cites.

Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts

Original

Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hannaneh Hajishirzi, Hao Cheng, Kai-Wei Chang, Michel Galley, and Jianfeng Gao. 2023 · 2023

Cited alongside, same era.

Modular visual question answering via code generation

Original

Sanjay Subramanian, Medhini Narasimhan, Kushal Khangaonkar, Kevin Yang, Arsha Nagrani, Cordelia Schmid, Andy Zeng, Trevor Darrell, and Dan Klein. 2023 · 2023

Cited alongside, same era.

Vipergpt: Visual inference via python execution for reasoning

Dídac Surís, Sachit Menon, and Carl Vondrick. 2023 · 2023

Cited alongside, same era.

Mm-vet: Evaluating large multimodal models for integrated capabilities

Original

Weihao Yu, Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Zicheng Liu, Xinchao Wang, and Lijuan Wang. 2023 · 2023

Cited alongside, same era.

Pixtral 12b

Discover the new multi-lingual, high-quality phi-3.5 slms

Microsoft. 2024 · 2024

Closest in time.

Compositional chain-of-thought prompting for large multimodal models

Chancharik Mitra, Brandon Huang, Trevor Darrell, and Roei Herzig. 2024 · 2024

Closest in time.

Vibe-eval: A hard evaluation suite for measuring progress of multimodal language models

Original

Piotr Padlewski, Max Bain, Matthew Henderson, Zhongkai Zhu, Nishant Relan, Hai Pham, Donovan Ong, Kaloyan Aleksiev, Aitor Ormazabal, Samuel Phua, et al. 2024 · 2024

Closest in time.

Vision language models are blind

Pooyan Rahmanzadehgervi, Logan Bolton, Mohammad Reza Taesiri, and Anh Totti Nguyen. 2024 · 2024

Closest in time.

Chartmimic: Evaluating lmm’s cross-modal reasoning capability via chart-to-code generation

Original

Chufan Shi, Cheng Yang, Yaxin Liu, Bo Shui, Junjie Wang, Mohan Jing, Linran Xu, Xinyu Zhu, Siheng Li, Yuxiang Zhang, et al. 2024 · 2024

Closest in time.