Fetching the paper…

Reading the bibliography…

2023

An Evaluation of GPT-4V and Gemini in Online VQA

Liu, Mengchen, Chen, Chongyan, Gurari, Danna

Original
PDF
Post on X

Understand

While there is much excitement about the potential of large multimodal models (LMM), a comprehensive evaluation is critical to establish their true capabilities and limitations.

In support of this aim, we evaluate two state-of-the-art LMMs, GPT-4V and Gemini, on a new visual question answering dataset sourced from an authentic online question answering community.
We conduct fine-grained analysis by generating seven types of metadata for nearly 2,000 visual questions, such as image type and the required image processing capabilities.
Our zero-shot performance analysis highlights the types of questions that are most challenging for both models, including questions related to "puzzling" topic, with "Identification" user intention, with "Sheet Music" image type, or labeled as "hard" by GPT-4.

An Evaluation of GPT-4V and Gemini in Online VQA · Around

Built on

Chain-of-thought prompting elicits reasoning in large language models
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al · 2022
Earlier work this paper cites.
Planning for AGI and beyond, 2023
OpenAI · 2023
Earlier work this paper cites.
GPT-4 technical report
Original
OpenAI · 2023
Earlier work this paper cites.
Gemini: A family of highly capable multimodal models, 2023
Google · 2023
Earlier work this paper cites.
A comprehensive evaluation of GPT-4V on knowledge-intensive visual question answering
Original
Yunxin Li, Longyue Wang, Baotian Hu, Xinyu Chen, Wanqi Zhong, Chenyang Lyu, and Min Zhang · 2023
Earlier work this paper cites.

Similar

On the road with GPT-4V (ision): Early explorations of visual-language model on autonomous driving
Original
Licheng Wen, Xuemeng Yang, Daocheng Fu, Xiaofeng Wang, Pinlong Cai, Xin Li, Tao Ma, Yingxuan Li, Linran Xu, Dengke Shang, et al · 2023
Cited alongside, same era.
The dawn of (lmms): Preliminary explorations with GPT-4V (ision)
Original
Zhengyuan Yang, Linjie Li, Kevin Lin, Jianfeng Wang, Chung-Ching Lin, Zicheng Liu, and Lijuan Wang · 2023
Cited alongside, same era.
A comprehensive study of GPT-4V’s multimodal capabilities in medical imaging
Yingshu Li, Yunyi Liu, Zhanyu Wang, Xinyu Liang, Lingqiao Liu, Lei Wang, Leyang Cui, Zhaopeng Tu, Longyue Wang, and Luping Zhou · 2023
Cited alongside, same era.
Fully authentic visual question answering dataset from online communities
Original
Chongyan Chen, Mengchen Liu, Noel Codella, Yunsheng Li, Lu Yuan, and Danna Gurari · 2023
Cited alongside, same era.
Levels of AGI: Operationalizing progress on the path to AGI

Then

GPT-4V Turbo API in Azure OpenAI Service, 2023
Microsoft · 2023
Closest in time.
Gemini api, 2023
Google · 2023
Closest in time.
MMMU: A massive multi-discipline multimodal understanding and reasoning benchmark for expert AGI
Original
Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, et al · 2023
Closest in time.
MathVista: Evaluating mathematical reasoning of foundation models in visual contexts
Original
Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hannaneh Hajishirzi, Hao Cheng, Kai-Wei Chang, Michel Galley, and Jianfeng Gao · 2023
Closest in time.
MM-Vet: Evaluating large multimodal models for integrated capabilities
Original
Weihao Yu, Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Zicheng Liu, Xinchao Wang, and Lijuan Wang · 2023
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Meredith Ringel Morris, Jascha Sohl-dickstein, Noah Fiedel, Tris Warkentin, Allan Dafoe, Aleksandra Faust, Clement Farabet, and Shane Legg · 2023

Cited alongside, same era.