Built on

Microsoft coco: Common objects in context
T.-Y. Lin et al · 2014
Earlier work this paper cites.
From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions
P. Young et al · 2014
Earlier work this paper cites.
Cider: Consensus-based image description evaluation
R. Vedantam et al · 2015
Earlier work this paper cites.
Microsoft coco captions: Data collection and evaluation server
Original
X. Chen et al · 2015
Earlier work this paper cites.
Clevr: A diagnostic dataset for compositional language and elementary visual reasoning, 2016
J. Johnson et al · 2016
Earlier work this paper cites.
Attention is all you need
A. Vaswani et al · 2017
Earlier work this paper cites.
Pointnet: Deep learning on point sets for 3d classification and segmentation
C. R. Qi et al · 2017
Earlier work this paper cites.
Making the v in vqa matter: Elevating the role of image understanding in visual question answering, 2017
Y. Goyal et al · 2017
Earlier work this paper cites.
Fvqa: Fact-based visual question answering
P. Wang et al · 2017
Earlier work this paper cites.
Scannet: Richly-annotated 3d reconstructions of indoor scenes
A. Dai et al · 2017
Earlier work this paper cites.
Vizwiz grand challenge: Answering visual questions from blind people
D. Gurari et al · 2018
Earlier work this paper cites.
A dataset of clinically generated visual questions and answers about radiology images
J. J. Lau et al · 2018
Earlier work this paper cites.
Gqa: A new dataset for real-world visual reasoning and compositional question answering, 2019
D. A. Hudson and C. D. Manning · 2019
Earlier work this paper cites.
Towards vqa models that can read
A. Singh et al · 2019
Earlier work this paper cites.
Ok-vqa: A visual question answering benchmark requiring external knowledge
K. Marino et al · 2019
Earlier work this paper cites.
Raven: A dataset for relational and analogical visual reasoning
C. Zhang et al · 2019
Earlier work this paper cites.
Lxmert: Learning cross-modality encoder representations from transformers, 2019
H. Tan and M. Bansal · 2019
Earlier work this paper cites.
Audiocaps: Generating captions for audios in the wild
C. D. Kim et al · 2019
Earlier work this paper cites.
Textcaps: a dataset for image captioning with reading comprehension
O. Sidorov et al · 2020
Earlier work this paper cites.
Document visual question answering challenge 2020
Original
M. Mathew et al · 2020
Earlier work this paper cites.
Point-bert: Pre-training 3d point cloud transformers with masked point modeling
Original
X. Yu et al · 2021
Earlier work this paper cites.
Learning transferable visual models from natural language supervision
A. Radford et al · 2021
Earlier work this paper cites.
Laion-400m: Open dataset of clip-filtered 400 million image-text pairs
Original
C. Schuhmann et al · 2021
Earlier work this paper cites.
Inter-gps: Interpretable geometry problem solving with formal language and symbolic reasoning
Original
P. Lu et al · 2021
Earlier work this paper cites.
Vilt: Vision-and-language transformer without convolution or region supervision, 2021
W. Kim et al · 2021
Earlier work this paper cites.
Multi-grained vision language pre-training: Aligning texts with visual concepts
Original
Y. Zeng et al · 2021
Earlier work this paper cites.
A-okvqa: A benchmark for visual question answering using world knowledge
D. Schwenk et al · 2022
Earlier work this paper cites.
Chartqa: A benchmark for question answering about charts with visual and logical reasoning
Original
A. Masry et al · 2022
Earlier work this paper cites.
Learn to explain: Multimodal reasoning via thought chains for science question answering
P. Lu et al · 2022
Earlier work this paper cites.
Scanqa: 3d question answering for spatial scene understanding
D. Azuma et al · 2022
Earlier work this paper cites.
Chain-of-thought prompting elicits reasoning in large language models
J. Wei et al · 2022
Earlier work this paper cites.
Seed-bench: Benchmarking multimodal llms with generative comprehension
Original
B. Li et al · 2023
Earlier work this paper cites.
A survey on large language models: Applications, challenges, limitations, and practical usage
M. U. Hadi et al · 2023
Earlier work this paper cites.
Llama: Open and efficient foundation language models
Original
H. Touvron et al · 2023
Earlier work this paper cites.
Judging llm-as-a-judge with mt-bench and chatbot arena, 2023
L. Zheng et al · 2023
Earlier work this paper cites.
Qlora: Efficient finetuning of quantized llms
Original
T. Dettmers et al · 2023
Earlier work this paper cites.
Visual spatial reasoning
F. Liu et al · 2023
Earlier work this paper cites.
Bubogpt: Enabling visual grounding in multi-modal llms
Original
Y. Zhao et al · 2023
Earlier work this paper cites.
Valley: Video assistant with large language model enhanced ability
Original
R. Luo et al · 2023
Earlier work this paper cites.
Robust speech recognition via large-scale weak supervision
A. Radford et al · 2023
Earlier work this paper cites.
Improved baselines with visual instruction tuning
Original
H. Liu et al · 2023
Earlier work this paper cites.
Aligning large multimodal models with factually augmented rlhf
Original
Z. Sun et al · 2023
Earlier work this paper cites.
Learning the user’s deeper preferences for multi-modal recommendation systems
F. Lei et al · 2023
Earlier work this paper cites.
Mmbench: Is your multi-modal model an all-around player?
Original
Y. Liu et al · 2023
Earlier work this paper cites.
Mm-vet: Evaluating large multimodal models for integrated capabilities
Original
W. Yu et al · 2023
Earlier work this paper cites.
Mme: A comprehensive evaluation benchmark for multimodal large language models
Original
C. Fu et al · 2023
Earlier work this paper cites.
Visual instruction tuning
Original
H. Liu et al · 2023
Earlier work this paper cites.
Equivariant similarity for vision-language foundation models
T. Wang et al · 2023
Earlier work this paper cites.
Touchstone: Evaluating vision-language models by language models
Original
S. Bai et al · 2023
Earlier work this paper cites.
mplug-owl2: Revolutionizing multi-modal large language model with modality collaboration
Original
Q. Ye et al · 2023
Earlier work this paper cites.
Vl-checklist: Evaluating pre-trained vision-language models with objects, attributes and relations, 2023
T. Zhao et al · 2023
Earlier work this paper cites.
When and why vision-language models behave like bags-of-words, and what to do about it?
M. Yuksekgonul et al · 2023
Earlier work this paper cites.
Q-bench: A benchmark for general-purpose foundation models on low-level vision
Original
H. Wu et al · 2023
Earlier work this paper cites.
On the hidden mystery of ocr in large multimodal models
Original
Y. Liu et al · 2023
Earlier work this paper cites.
Hierarchical multimodal transformers for multipage docvqa
R. Tito et al · 2023
Earlier work this paper cites.
Contextual object detection with multimodal large language models
Original
Y. Zang et al · 2023
Earlier work this paper cites.
What’s” up” with vision-language models? investigating their struggle with spatial reasoning
Original
A. Kamath et al · 2023
Earlier work this paper cites.
Chartbench: A benchmark for complex visual reasoning in charts
Original
Z. Xu et al · 2023
Earlier work this paper cites.
Scigraphqa: A large-scale synthetic multi-turn question-answering dataset for scientific graphs
Original
S. Li and N. Tajbakhsh · 2023
Earlier work this paper cites.
Mmc: Advancing multimodal chart understanding with large-scale instruction tuning
Original
F. Liu et al · 2023
Earlier work this paper cites.
Chef: A comprehensive evaluation framework for standardized assessment of multimodal large language models
Original
Z. Shi et al · 2023
Earlier work this paper cites.
Benchlmm: Benchmarking cross-style visual capability of large multimodal models
Original
R. Cai et al · 2023
Earlier work this paper cites.

Similar

Evaluating object hallucination in large vision-language models
Original
Y. Li et al · 2023
Cited alongside, same era.
Mitigating hallucination in large multi-modal models via robust instruction tuning
Original
F. Liu et al · 2023
Cited alongside, same era.
Evaluation and analysis of hallucination in large vision-language models
Original
J. Wang et al · 2023
Cited alongside, same era.
Holistic analysis of hallucination in gpt-4v (ision): Bias and interference challenges
Original
C. Cui et al · 2023
Cited alongside, same era.
An llm-free multi-dimensional benchmark for mllms hallucination evaluation
Original

Then

Vlkeb: A large vision-language model knowledge editing benchmark, 2024
H. Huang et al · 2024
Closest in time.
Mc-mke: A fine-grained multimodal knowledge editing benchmark emphasizing modality consistency
Original
J. Zhang et al · 2024
Closest in time.
Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi
X. Yue et al · 2024
Closest in time.
Mementos: A comprehensive benchmark for multimodal large language model reasoning over image sequences
Original
X. Wang et al · 2024
Closest in time.
Marvel: Multidimensional abstraction and reasoning through visual evaluation and learning
Original
Y. Jiang et al · 2024
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…

2024

A Survey on Evaluation of Multimodal Large Language Models

Huang, Jiaxing, Zhang, Jingyi

Understand

Multimodal Large Language Models (MLLMs) mimic human perception and reasoning system by integrating powerful Large Language Models (LLMs) with various modality encoders (e.g., vision, audio), positioning LLMs as the "brain" and various modality encoders as sensory organs.

This framework endows MLLMs with human-like capabilities, and suggests a potential pathway towards achieving artificial general intelligence (AGI).
With the emergence of all-round MLLMs like GPT-4V and Gemini, a multitude of evaluation methods have been developed to assess their capabilities across different dimensions.
This paper presents a systematic and comprehensive review of MLLM evaluation methods, covering the following key aspects: (1) the background of MLLMs and their evaluation; (2) "what to evaluate" that reviews and categorizes existing MLLM evaluation tasks based on the capabilities assessed, including general multimodal recognition, perception, reasoning and trustworthiness, and domain-specific applications such as socioeconomic, natural sciences and engineering, medical usage, AI agent, remote sensing, video and audio processing, 3D point cloud analysis, and others; (3) "where to evaluate" that summarizes MLLM evaluation benchmarks into general and specific benchmarks; (4) "how to evaluate" that reviews and illustrates MLLM evaluation steps and metrics; Our overarching goal is to provide valuable insights for researchers in the field of MLLM evaluation, thereby facilitating the development of more capable and reliable MLLMs.