Fetching the paper…

Reading the bibliography…

2024

A Survey on Multimodal Benchmarks: In the Era of Large AI Models

Li, Lin, Chen, Guikun, Shi, Hanrong et al.

Original
PDF
Post on X

Understand

The rapid evolution of Multimodal Large Language Models (MLLMs) has brought substantial advancements in artificial intelligence, significantly enhancing the capability to understand and generate multimodal content.

While prior studies have largely concentrated on model architectures and training methodologies, a thorough analysis of the benchmarks used for evaluating these models remains underexplored.
This survey addresses this gap by systematically reviewing 211 benchmarks that assess MLLMs across four core domains: understanding, reasoning, generation, and application.
We provide a detailed analysis of task designs, evaluation metrics, and dataset constructions, across diverse modalities.

A Survey on Multimodal Benchmarks: In the Era of Large AI Models · Around

Built on

J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei, “Imagenet: A large-scale hierarchical image database,” in
2009
Earlier work this paper cites.
T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár, and C. L. Zitnick, “Microsoft coco: Common objects in context,” in
2014
Earlier work this paper cites.
T. Salimans, I. Goodfellow, W. Zaremba, V. Cheung, A. Radford, and X. Chen, “Improved techniques for training gans,” in
2016
Earlier work this paper cites.
R. Krishna, Y. Zhu, O. Groth, J. Johnson, K. Hata, J. Kravitz, S. Chen, Y. Kalantidis, L.-J. Li, D. A. Shamma
2017
Earlier work this paper cites.
M. Balog, A. Gaunt, M. Brockschmidt, S. Nowozin, and D. Tarlow, “Deepcoder: Learning to write programs,” in
2017
Earlier work this paper cites.
M. Heusel, H. Ramsauer, T. Unterthiner, B. Nessler, and S. Hochreiter, “Gans trained by a two time-scale update rule converge to a local nash equilibrium,” in
2017
Earlier work this paper cites.
A. Rohrbach, L. A. Hendricks, K. Burns, T. Darrell, and K. Saenko, “Object hallucination in image captioning,” in
2018
Earlier work this paper cites.
X. Li, X. Yin, C. Li, P. Zhang, X. Hu, L. Zhang, L. Wang, H. Hu, L. Dong, F. Wei
2020
Earlier work this paper cites.
M. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. D. O. Pinto, J. Kaplan, H. Edwards, Y. Burda, N. Joseph, G. Brockman
Original
2021
Earlier work this paper cites.
OpenAI, “OpenAI: Introducing ChatGPT,” 2022. [Online]. Available:
2022
Earlier work this paper cites.
L. Fan, G. Wang, Y. Jiang, A. Mandlekar, Y. Yang, H. Zhu, A. Tang, D.-A. Huang, Y. Zhu, and A. Anandkumar, “Minedojo: Building open-ended embodied agents with internet-scale knowledge,” in
2022
Earlier work this paper cites.
J.-B. Alayrac, J. Donahue, P. Luc, A. Miech, I. Barr, Y. Hasson, K. Lenc, A. Mensch, K. Millican, M. Reynolds
2022
Earlier work this paper cites.
Y. Li, W. Li, and L. Nie, “Mmcoqa: Conversational question answering over text, tables, and images,” in
2022
Earlier work this paper cites.
S. Lin, J. Hilton, and O. Evans, “Truthfulqa: Measuring how models mimic human falsehoods,” in
2022
Earlier work this paper cites.
C. Schuhmann, R. Beaumont, R. Vencu, C. Gordon, R. Wightman, M. Cherti, T. Coombes, A. Katta, C. Mullis, M. Wortsman
2022
Earlier work this paper cites.
J. Achiam, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, F. L. Aleman, D. Almeida, J. Altenschmidt, S. Altman, S. Anadkat
Original
2023
Earlier work this paper cites.
G. Team, R. Anil, S. Borgeaud, Y. Wu, J.-B. Alayrac, J. Yu, R. Soricut, J. Schalkwyk, A. M. Dai, A. Hauth
Original
2023
Earlier work this paper cites.
G. Wang, Y. Ge, X. Ding, M. Kankanhalli, and Y. Shan, “What makes for good visual tokenizers for large language models?”
Original
2023
Earlier work this paper cites.
Y. Zang, W. Li, J. Han, K. Zhou, and C. C. Loy, “Contextual object detection with multimodal large language models,”
Original
2023
Earlier work this paper cites.
T. Wang, K. Lin, L. Li, C.-C. Lin, Z. Yang, H. Zhang, Z. Liu, and L. Wang, “Equivariant similarity for vision-language foundation models,” in
2023
Earlier work this paper cites.
B. Li, P. Zhang, J. Yang, Y. Zhang, F. Pu, and Z. Liu, “Otterhd: A high-resolution multi-modality model,”
Original
2023
Earlier work this paper cites.
P. Xu, W. Shao, K. Zhang, P. Gao, S. Liu, M. Lei, F. Meng, S. Huang, Y. Qiao, and P. Luo, “Lvlm-ehub: A comprehensive evaluation benchmark for large vision-language models,”
Original
2023
Earlier work this paper cites.
Q. Ye, H. Xu, G. Xu, J. Ye, M. Yan, Y. Zhou, J. Wang, A. Hu, P. Shi, Y. Shi
Original
2023
Earlier work this paper cites.
Y. Liu, H. Duan, Y. Zhang, B. Li, S. Zhang, W. Zhao, Y. Yuan, J. Wang, C. He, Z. Liu
Original
2023
Earlier work this paper cites.
B. Li, R. Wang, G. Wang, Y. Ge, Y. Ge, and Y. Shan, “Seed-bench: Benchmarking multimodal llms with generative comprehension,”
Original
2023
Earlier work this paper cites.
S. Bai, S. Yang, J. Bai, P. Wang, X. Zhang, J. Lin, X. Wang, C. Zhou, and J. Zhou, “Touchstone: Evaluating vision-language models by language models,”
Original
2023
Earlier work this paper cites.
Z. Shi, Z. Wang, H. Fan, Z. Yin, L. Sheng, Y. Qiao, and J. Shao, “Chef: A comprehensive evaluation framework for standardized assessment of multimodal large language models,”
Original
2023
Earlier work this paper cites.
B. Wang, Z. Liu, X. Huang, F. Jiao, Y. Ding, A. T. Aw, and N. F. Chen, “Seaeval for multilingual foundation models: From cross-lingual alignment to cultural reasoning,”
Original
2023
Earlier work this paper cites.
S. Li, L. Li, S. Ren, Y. Liu, Y. Liu, R. Gao, X. Sun, and L. Hou, “Vitatecs: A diagnostic dataset for temporal concept understanding of video-language models,”
Original
2023
Earlier work this paper cites.
X. Chen, Y. Lin, Y. Zhang, and W. Huang, “Autoeval-video: An automatic benchmark for assessing large vision language models in open-ended video question answering,”
Original
2023
Earlier work this paper cites.
M. Ning, B. Zhu, Y. Xie, B. Lin, J. Cui, L. Yuan, D. Chen, and L. Yuan, “Video-bench: A comprehensive benchmark and toolkit for evaluating video-based large language models,”
Original
2023
Earlier work this paper cites.
M. Li, X. Chen, C. Zhang, S. Chen, H. Zhu, F. Yin, G. Yu, and T. Chen, “M3dbench: Let’s instruct large models with multi-modal 3d prompts,”
Original
2023
Earlier work this paper cites.
W. Zhang, M. Aljunied, C. Gao, Y. K. Chia, and L. Bing, “M3exam: A multilingual, multimodal, multilevel benchmark for examining large language models,” in
2023
Earlier work this paper cites.
J. Van Landeghem, R. Tito, Ł. Borchmann, M. Pietruszka, P. Joziak, R. Powalski, D. Jurkiewicz, M. Coustaty, B. Anckaert, E. Valveny
2023
Earlier work this paper cites.
Z. Xu, S. Du, Y. Qi, C. Xu, C. Yuan, and J. Guo, “Chartbench: A benchmark for complex visual reasoning in charts,”
Original
2023
Earlier work this paper cites.
S. Li and N. Tajbakhsh, “Scigraphqa: A large-scale synthetic multi-turn question-answering dataset for scientific graphs,”
Original
2023
Earlier work this paper cites.
A. Kamath, J. Hessel, and K.-W. Chang, “What’s “up” with vision-language models? investigating their struggle with spatial reasoning,” in
2023
Earlier work this paper cites.
I. Kesen, A. Pedrotti, M. Dogan, M. Cafagna, E. C. Acikgoz, L. Parcalabescu, I. Calixto, A. Frank, A. Gatt, A. Erdem
Original
2023
Earlier work this paper cites.
S. Cheng, B. Tian, Q. Liu, X. Chen, Y. Wang, H. Chen, and N. Zhang, “Can we edit multimodal large language models?” in
2023
Earlier work this paper cites.
J. An, Z. Yang, L. Li, J. Wang, K. Lin, Z. Liu, L. Wang, and J. Luo, “Openleaf: Open-domain interleaved image-text generation and evaluation,”
Original
2023
Earlier work this paper cites.
J. Li, K. Pan, Z. Ge, M. Gao, W. Ji, W. Zhang, T.-S. Chua, S. Tang, H. Zhang, and Y. Zhuang, “Fine-tuning multimodal llms to follow zero-shot demonstrative instructions,” in
2023
Earlier work this paper cites.
Y. Bitton, H. Bansal, J. Hessel, R. Shao, W. Zhu, A. Awadalla, J. Gardner, R. Taori, and L. Schimdt, “Visit-bench: a benchmark for vision-language instruction following inspired by real-world use,” in
2023
Earlier work this paper cites.
C. Cui, Y. Zhou, X. Yang, S. Wu, L. Zhang, J. Zou, and H. Yao, “Holistic analysis of hallucination in gpt-4v (ision): Bias and interference challenges,”
Original
2023
Earlier work this paper cites.
F. Liu, K. Lin, L. Li, J. Wang, Y. Yacoob, and L. Wang, “Mitigating hallucination in large multi-modal models via robust instruction tuning,” in
2023
Earlier work this paper cites.
Z. Sun, S. Shen, S. Cao, H. Liu, C. Li, Y. Shen, C. Gan, L.-Y. Gui, Y.-X. Wang, Y. Yang
Original
2023
Earlier work this paper cites.
J. Wang, Y. Wang, G. Xu, J. Zhang, Y. Gu, H. Jia, M. Yan, J. Zhang, and J. Sang, “An llm-free multi-dimensional benchmark for mllms hallucination evaluation,”
Original
2023
Earlier work this paper cites.
H. Lovenia, W. Dai, S. Cahyawijaya, Z. Ji, and P. Fung, “Negative object presence evaluation (nope) to measure object hallucination in vision-language models,”
Original
2023
Earlier work this paper cites.
J. Wang, Y. Zhou, G. Xu, P. Shi, C. Zhao, H. Xu, Q. Ye, M. Yan, J. Zhang, J. Zhu
Original
2023
Earlier work this paper cites.
Y. Li, Y. Du, K. Zhou, J. Wang, X. Zhao, and J.-R. Wen, “Evaluating object hallucination in large vision-language models,” in
2023
Earlier work this paper cites.
X. Liu, Y. Zhu, J. Gu, Y. Lan, C. Yang, and Y. Qiao, “Mm-safetybench: A benchmark for safety evaluation of multimodal large language models,”
Original
2023
Earlier work this paper cites.
R. Cai, Z. Song, D. Guan, Z. Chen, X. Luo, C. Yi, and A. Kot, “Benchlmm: Benchmarking cross-style visual capability of large multimodal models,”
Original
2023
Earlier work this paper cites.
L. Chen, Y. Zhang, S. Ren, H. Zhao, Z. Cai, Y. Wang, P. Wang, T. Liu, and B. Chang, “Towards end-to-end embodied decision making via multi-modal large language model: Explorations with gpt4-vision and beyond,” 2023
2023
Earlier work this paper cites.
W. Dai, J. Li, D. LI, A. Tiong, J. Zhao, W. Wang, B. Li, P. N. Fung, and S. Hoi, “Instructblip: Towards general-purpose vision-language models with instruction tuning,” in
2023
Earlier work this paper cites.
O. Contributors, “Opencompass: A universal evaluation platform for foundation models,”
2023
Earlier work this paper cites.
J. Zhou, T. Lu, S. Mishra, S. Brahma, S. Basu, Y. Luan, D. Zhou, and L. Hou, “Instruction-following evaluation for large language models,”
Original
2023
Earlier work this paper cites.
Y. Li, Y. Du, K. Zhou, J. Wang, X. Zhao, and J.-R. Wen, “Evaluating object hallucination in large vision-language models,” in
2023
Earlier work this paper cites.
Z. Ji, N. Lee, R. Frieske, T. Yu, D. Su, Y. Xu, E. Ishii, Y. J. Bang, A. Madotto, and P. Fung, “Survey of hallucination in natural language generation,”
2023
Earlier work this paper cites.
S. Yin, C. Fu, S. Zhao, K. Li, X. Sun, T. Xu, and E. Chen, “A survey on multimodal large language models,”
Original
2023
Earlier work this paper cites.
D. Kelly, Y. Chen, S. E. Cornwell, N. S. Delellis, A. Mayhew, S. Onaolapo, and V. L. Rubin, “Bing chat: the future of search engines?”
2023
Earlier work this paper cites.
J. Betker, G. Goh, L. Jing, T. Brooks, J. Wang, L. Li, L. Ouyang, J. Zhuang, J. Lee, Y. Guo
2023
Earlier work this paper cites.
A. K. M. Abbas, K. Tirumala, D. Simig, S. Ganguli, and A. S. Morcos, “Semdedup: Data-efficient learning at web-scale through semantic deduplication,” in
2023
Earlier work this paper cites.
S. Y. Gadre, G. Ilharco, A. Fang, J. Hayase, G. Smyrnis, T. Nguyen, R. Marten, M. Wortsman, D. Ghosh, J. Zhang
2023
Earlier work this paper cites.
Z. Zhang, H. Wu, E. Zhang, G. Zhai, and W. Lin, “Q-bench: A benchmark for multi-modal foundation models on low-level vision from single images to pairs,”
2024
Earlier work this paper cites.
W. Peng, S. Xie, Z. You, S. Lan, and Z. Wu, “Synthesize diagnose and optimize: Towards fine-grained vision-language understanding,” in
2024
Earlier work this paper cites.
P. Wu and S. Xie, “V⁢: Guided visual search as a core mechanism in multimodal llms,” in
2024
Earlier work this paper cites.
Y. Liu, Z. Li, M. Huang, B. Yang, W. Yu, C. Li, X. Yin, C. lin Liu, L. Jin, and X. Bai, “On the hidden mystery of ocr in large multimodal models,”
Original
2024
Earlier work this paper cites.
Y. Wang, Y. Liao, H. Liu, H. Liu, Y. Wang, and Y. Wang, “Mm-sap: A comprehensive benchmark for assessing self-awareness of multimodal large language models in perception,”
Original
2024
Earlier work this paper cites.
S. Tong, Z. Liu, Y. Zhai, Y. Ma, Y. LeCun, and S. Xie, “Eyes wide shut? exploring the visual shortcomings of multimodal llms,” in
2024
Earlier work this paper cites.
S. Tong, E. Brown, P. Wu, S. Woo, M. Middepogu, S. C. Akula, J. Yang, S. Yang, A. Iyer, X. Pan, A. Wang, R. Fergus, Y. LeCun, and S. Xie, “Cambrian-1: A fully open, vision-centric exploration of multimodal llms,”
Original
2024
Earlier work this paper cites.
J. Chen, Y. Liu, D. Li, X. An, W. Deng, Z. Feng, Y. Zhao, and Y. Xie, “Plug-and-play grounding of reasoning in multimodal large language models,”
Original
2024
Earlier work this paper cites.
W. Lin, X. Wei, R. An, P. Gao, B. Zou, Y. Luo, S. Huang, S. Zhang, and H. Li, “Draw-and-understand: Leveraging visual prompts to enable mllms to comprehend what you want,”
Original
2024
Earlier work this paper cites.
J. Li, Q. Wei, C. Zhang, G. Qi, M. Du, Y. Chen, and S. Bi, “Single image unlearning: Efficient machine unlearning in multimodal large language models,”
Original
2024
Earlier work this paper cites.
Z. Zhou, Q. Wang, B. Lin, Y. Su, R. Chen, X. Tao, A. Zheng, L. Yuan, P. Wan, and D. Zhang, “Uniaa: A unified multi-modal image aesthetic assessment baseline and benchmark,”
Original
2024
Earlier work this paper cites.
Y. Huang, Q. Yuan, X. Sheng, Z. Yang, H. Wu, P. Chen, Y. Yang, L. Li, and W. Lin, “Aesbench: An expert benchmark for multimodal large language models on image aesthetics perception,”
Original
2024
Earlier work this paper cites.
Z. Liu, F. Fang, X. Feng, X. Du, C. Zhang, Z. Wang, Y. Bai, Q. Zhao, L. Fan, C. Gan
Original
2024
Earlier work this paper cites.
H. P. Zou, V. Samuel, Y. Zhou, W. Zhang, L. Fang, Z. Song, P. S. Yu, and C. Caragea, “Implicitave: An open-source dataset and multimodal llms benchmark for implicit attribute value extraction,” in
2024
Earlier work this paper cites.
Q. Yang, M. Ye, and B. Du, “Emollm: Multimodal emotional understanding meets large language models,”
Original
2024
Earlier work this paper cites.
Y. Li, A. Dao, W. Bao, Z. Tan, T. Chen, H. Liu, and Y. Kong, “Facial affective behavior analysis with instruction tuning,”
Original
2024
Earlier work this paper cites.
W. Shao, M. Lei, Y. Hu, P. Gao, K. Zhang, F. Meng, P. Xu, S. Huang, H. Li, Y. Qiao, and P. Luo, “Tinylvlm-ehub: Towards comprehensive and efficient evaluation for large vision-language models,”
Original
2024
Earlier work this paper cites.

Similar

Z. Yin, J. Wang, J. Cao, Z. Shi, D. Liu, M. Li, X. Huang, Z. Wang, L. Sheng, L. Bai
2024
Cited alongside, same era.
C. Fu, P. Chen, Y. Shen, Y. Qin, M. Zhang, X. Lin, J. Yang, X. Zheng, K. Li, X. Sun, Y. Wu, and R. Ji, “Mme: A comprehensive evaluation benchmark for multimodal large language models,”
Original
2024
Cited alongside, same era.
R. Pi, J. Zhang, J. Zhang, R. Pan, Z. Chen, and T. Zhang, “Image textualization: An automatic framework for creating accurate and detailed image descriptions,”
Original
2024
Cited alongside, same era.
B. Li, Y. Ge, Y. Ge, G. Wang, R. Wang, R. Zhang, and Y. Shan, “Seed-bench-2: Benchmarking multimodal large language models,” in
2024
Cited alongside, same era.
B. Li, Y. Ge, Y. Chen, Y. Ge, R. Zhang, and Y. Shan, “Seed-bench-2-plus: Benchmarking multimodal large language models with text-rich visual comprehension,”
Original

Then

X. Cao, B. Lai, W. Ye, Y. Ma, J. Heintz, J. Chen, J. Cao, and J. M. Rehg, “What is the visual cognition gap between humans and multimodal llms?”
Original
2024
Closest in time.
A.-C. Cheng, H. Yin, Y. Fu, Q. Guo, R. Yang, J. Kautz, X. Wang, and S. Liu, “Spatialrgpt: Grounded spatial reasoning in vision language model,”
Original
2024
Closest in time.
Y. Jiang, J. Zhang, K. Sun, Z. Sourati, K. Ahrabian, K. Ma, F. Ilievski, and J. Pujara, “Marvel: Multidimensional abstraction and reasoning through visual evaluation and learning,”
Original
2024
Closest in time.
H. Shao, S. Qian, H. Xiao, G. Song, Z. Zong, L. Wang, Y. Liu, and H. Li, “Visual cot: Advancing multi-modal language models with a comprehensive dataset and benchmark for chain-of-thought reasoning,”
Original
2024
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

2024

Cited alongside, same era.

X. Fu, Y. Hu, B. Li, Y. Feng, H. Wang, X. Lin, D. Roth, N. A. Smith, W.-C. Ma, and R. Krishna, “Blink: Multimodal large language models can see but not perceive,”

2024

Cited alongside, same era.

Y. Zeng, H. Zhang, J. Zheng, J. Xia, G. Wei, Y. Wei, Y. Zhang, T. Kong, and R. Song, “What matters in training a gpt4-style language model with multimodal inputs?” in

2024

Cited alongside, same era.

H. Al-Tahan, Q. Garrido, R. Balestriero, D. Bouchacourt, C. Hazirbas, and M. Ibrahim, “Unibench: Visual reasoning requires rethinking vision-language beyond scaling,”

2024

Cited alongside, same era.

X. Li, F. Zhang, H. Diao, Y. Wang, X. Wang, and L.-Y. Duan, “Densefusion-1m: Merging vision experts for comprehensive multimodal perception,”

2024

Cited alongside, same era.

L. Chen, J. Li, X. Dong, P. Zhang, Y. Zang, Z. Chen, H. Duan, J. Wang, Y. Qiao, D. Lin, and F. Zhao, “Are we on the right way for evaluating large vision-language models?”

Original

2024

Cited alongside, same era.

C. Tran and H. L. Thanh, “Lavy: Vietnamese multimodal large language model,”

2024

Cited alongside, same era.

H.-L. Sun, D.-W. Zhou, Y. Li, S. Lu, C. Yi, Q.-G. Chen, Z. Xu, W. Luo, K. Zhang, D.-C. Zhan

2024

Cited alongside, same era.

Y. Xiao, E. Sun, T. Liu, and W. Wang, “Logicvista: Multimodal llm logical reasoning benchmark in visual contexts,”

2024

Closest in time.

Y. Wang, Y. Zeng, J. Zheng, X. Xing, J. Xu, and X. Xu, “Videocot: A video chain-of-thought dataset with active annotation tool,”

2024

Closest in time.

Y. Li, W. Tian, Y. Jiao, J. Chen, and Y.-G. Jiang, “Eyes can deceive: Benchmarking counterfactual reasoning abilities of multi-modal large language models,”

2024

Closest in time.

J. Zhang, H. Zhang, X. Yin, B. Huang, X. Zhang, X. Hu, and X. Wan, “Mc-mke: A fine-grained multimodal knowledge editing benchmark emphasizing modality consistency,”

Original

2024

Closest in time.

H. Huang, H. Zhong, T. Yu, Q. Liu, S. Wu, L. Wang, and T. Tan, “Vlkeb: A large vision-language model knowledge editing benchmark,”

2024

Closest in time.

J. Li, M. Du, C. Zhang, Y. Chen, N. Hu, G. Qi, H. Jiang, S. Cheng, and B. Tian, “Mike: A new benchmark for fine-grained multimodal entity knowledge editing,”

2024

Closest in time.

H. Hengyuan Zhao, P. Zhou, D. Gao, and M. Z. Shou, “Lova3: Learning to visual question answering, asking and assessment,”

2024

Closest in time.

C. Liu, H. Wu, Y. Zhong, X. Zhang, Y. Wang, and W. Xie, “Intelligent grimm-open-ended visual storytelling via latent diffusion models,” in

2024

Closest in time.

S. Yang, Y. Ge, Y. Li, Y. Chen, Y. Ge, Y. Shan, and Y. Chen, “Seed-story: Multimodal long story generation with large language model,”

2024

Closest in time.

S. Yun, H. Lin, R. Thushara, M. Q. Bhat, Y. Wang, Z. Jiang, M. Deng, J. Wang, T. Tao, J. Li

2024

Closest in time.

C. Wu, Y. Ge, Q. Guo, J. Wang, Z. Liang, Z. Lu, Y. Shan, and P. Luo, “Plot2code: A comprehensive benchmark for evaluating multi-modal large language models in code generation from scientific plots,”

2024

Closest in time.

C. Chen, J. Zhu, X. Luo, H. Shen, L. Gao, and J. Song, “Coin: A benchmark of continual instruction tuning for multimodel large language model,”

2024

Closest in time.

Y. Qian, H. Ye, J.-P. Fauconnier, P. Grasch, Y. Yang, and Z. Gan, “Mia-bench: Towards better instruction following evaluation of multimodal llms,”

2024

Closest in time.

H. Liu, C. Li, Q. Wu, and Y. J. Lee, “Visual instruction tuning,” in

2024

Closest in time.

A. Gunjal, J. Yin, and E. Bas, “Detecting and preventing hallucinations in large vision language models,” in

2024

Closest in time.

L. Cao, V. Buchner, Z. Senane, and F. Yang, “GenCeption: Evaluate multimodal llms with unlabeled unimodal data,”

2024

Closest in time.

W. Huang, H. Liu, M. Guo, and N. Z. Gong, “Visual hallucinations of multi-modal large language models,”

2024

Closest in time.

Y. Qian, H. Zhang, Y. Yang, and Z. Gan, “How easy is it to fool your multimodal llms? an empirical analysis on deceptive prompts,”

2024

Closest in time.

S. Cha, J. Lee, Y. Lee, and C. Yang, “Visually dehallucinative instruction generation,” in

2024

Closest in time.

X. Chen, C. Wang, Y. Xue, N. Zhang, X. Yang, Q. Li, Y. Shen, J. Gu, and H. Chen, “Unified hallucination detection for multimodal large language models,”

2024

Closest in time.

T. Guan, F. Liu, X. Wu, R. Xian, Z. Li, X. Liu, X. Wang, L. Chen, F. Huang, Y. Yacoob

2024

Closest in time.

K. Zheng, J. Chen, Y. Yan, X. Zou, and X. Hu, “Reefknot: A comprehensive benchmark for relation hallucination evaluation, analysis and mitigation in multimodal large language models,”

2024

Closest in time.

P. Ding, J. Wu, J. Kuang, D. Ma, X. Cao, X. Cai, S. Chen, J. Chen, and S. Huang, “Hallu-pi: Evaluating hallucination in multi-modal large language models within perturbed inputs,” in

2024

Closest in time.

Z. Wang, G. Bingham, A. Yu, Q. Le, T. Luong, and G. Ghiasi, “Haloquest: A visual hallucination dataset for advancing multimodal reasoning,”

2024

Closest in time.

M. Ye-Bin, N. Hyeon-Woo, W. Choi, and T.-H. Oh, “Beaf: Observing before-after changes to evaluate hallucination in vision-language models,”

2024

Closest in time.

X. Chen, Z. Ma, X. Zhang, S. Xu, S. Qian, J. Yang, D. F. Fouhey, and J. Chai, “Multi-object hallucination in vision-language models,”

2024

Closest in time.

B. Yan, J. Zhang, Z. Yuan, S. Shan, and X. Chen, “Evaluating the quality of hallucination benchmarks for large vision-language models,”

2024

Closest in time.

Z. Meng, Y. Dai, Z. Gong, S. Guo, M. Tang, and T. Wei, “Vga: Vision gui assistant–minimizing hallucinations through image-centric fine-tuning,”

2024

Closest in time.

S. Wang, H. Lin, Z. Luo, Z. Ye, G. Chen, and J. Ma, “Mfc-bench: Benchmarking multimodal fact-checking with large vision-language models,”

2024

Closest in time.

X. Wu, T. Guan, D. Li, S. Huang, X. Liu, X. Wang, R. Xian, A. Shrivastava, F. Huang, J. L. Boyd-Graber

2024

Closest in time.

Y. Wang, Y. Wang, D. Zhao, C. Xie, and Z. Zheng, “Videohallucer: Evaluating intrinsic and extrinsic hallucinations in large video-language models,”

2024

Closest in time.

P. Kaul, Z. Li, H. Yang, Y. Dukler, A. Swaminathan, C. Taylor, and S. Soatto, “Throne: An object-based hallucination benchmark for the free-form generations of large vision-language models,” in

2024

Closest in time.

J. Chen, D. Yang, T. Wu, Y. Jiang, X. Hou, M. Li, S. Wang, D. Xiao, K. Li, and L. Zhang, “Detecting and evaluating medical hallucinations in large vision language models,”

2024

Closest in time.

L. Fieback, J. Spiegelberg, and H. Gottschalk, “Metatoken: Detecting hallucination in image descriptions by meta classification,”

2024

Closest in time.

M. Zhang and K. Rong, “Automated multi-level preference for mllms,”

2024

Closest in time.

X. Li, H. Zhou, R. Wang, T. Zhou, M. Cheng, and C.-J. Hsieh, “Mossbench: Is your multimodal language model oversensitive to safe queries?”

2024

Closest in time.

T. Gu, Z. Zhou, K. Huang, D. Liang, Y. Wang, H. Zhao, Y. Yao, X. Qiao, K. Wang, Y. Yang

2024

Closest in time.

M. Li, L. Li, Y. Yin, M. Ahmed, Z. Liu, and Q. Liu, “Red teaming visual language models,”

2024

Closest in time.

Y. Zhang, Y. Huang, Y. Sun, C. Liu, Z. Zhao, Z. Fang, Y. Wang, H. Chen, X. Yang, X. Wei, H. Su, Y. Dong, and J. Zhu, “Benchmarking trustworthiness of multimodal large language models: A comprehensive study,”

2024

Closest in time.

W. Liu, N. Wu, W. Ding, S. Liang, M. Gong, and D. Zhang, “Towards truthful multilingual large language models: Benchmarking and alignment strategies,”

2024

Closest in time.

Y. Shi, Y. Gao, Y. Lai, H. Wang, J. Feng, L. He, J. Wan, C. Chen, Z. Yu, and X. Cao, “Shield: An evaluation benchmark for face spoofing and forgery detection with multimodal large language models,”

2024

Closest in time.

W. Luo, S. Ma, X. Liu, X. Guo, and C. Xiao, “Jailbreakv-28k: A benchmark for assessing the robustness of multimodal large language models against jailbreak attacks,”

2024

Closest in time.

Y. Liu, Z. Liang, Y. Wang, M. He, J. Li, and B. Zhao, “Seeing clearly, answering incorrectly: A multimodal robustness benchmark for evaluating mllms on leading questions,”

2024

Closest in time.

J. Zhang, T. Pang, C. Du, Y. Ren, B. Li, and M. Lin, “Benchmarking large multimodal models against common corruptions,”

2024

Closest in time.

T. Han, Q. Lian, R. Pan, R. Pi, J. Zhang, S. Diao, Y. Lin, and T. Zhang, “The instinctive bias: Spurious images lead to hallucination in mllms,”

2024

Closest in time.

W. Ye, G. Zheng, Y. Ma, X. Cao, B. Lai, J. M. Rehg, and A. Zhang, “Mm-spubench: Towards better understanding of spurious biases in multimodal llms,”

2024

Closest in time.

X. Deng, Y. Gu, B. Zheng, S. Chen, S. Stevens, B. Wang, H. Sun, and Y. Su, “Mind2web: Towards a generalist agent for the web,” in

2024

Closest in time.

C. Rawles, A. Li, D. Rodriguez, O. Riva, and T. Lillicrap, “Androidinthewild: A large-scale dataset for android device control,” in

2024

Closest in time.

J. Y. Koh, R. Lo, L. Jang, V. Duvvur, M. C. Lim, P.-Y. Huang, G. Neubig, S. Zhou, R. Salakhutdinov, and D. Fried, “Visualwebarena: Evaluating multimodal agents on realistic visual web tasks,” in

2024

Closest in time.

T. Xu, L. Chen, D.-J. Wu, Y. Chen, Z. Zhang, X. Yao, Z. Xie, Y. Chen, S. Liu, B. Qian

2024

Closest in time.

K. You, H. Zhang, E. Schoop, F. Weers, A. Swearngin, J. Nichols, Y. Yang, and Z. Gan, “Ferret-ui: Grounded mobile ui understanding with multimodal llms,”

2024

Closest in time.

J. Wang, H. Xu, J. Ye, M. Yan, W. Shen, J. Zhang, F. Huang, and J. Sang, “Mobile-agent: Autonomous multi-modal mobile device agent with visual perception,” in

2024

Closest in time.

Y. Fan, L. Ding, C.-C. Kuo, S. Jiang, Y. Zhao, X. Guan, J. Yang, Y. Zhang, and X. E. Wang, “Read anywhere pointed: Layout-aware gui screen reading with tree-of-lens grounding,”

2024

Closest in time.

Y. Chen, Y. Ge, Y. Ge, M. Ding, B. Li, R. Wang, R. Xu, Y. Shan, and X. Liu, “Egoplan-bench: Benchmarking multimodal large language models for human-level planning,”

2024

Closest in time.

A. Majumdar, A. Ajay, X. Zhang, P. Putta, S. Yenamandra, M. Henaff, S. Silwal, P. Mcvay, O. Maksymets, S. Arnaud, K. Yadav, Q. Li, B. Newman, M. Sharma, V. Berges, S. Zhang, P. Agrawal, Y. Bisk, D. Batra, M. Kalakrishnan, F. Meier, C. Paxton, S. Sax, and A. Rajeswaran, “Openeqa: Embodied question answering in the era of foundation models,” in

2024

Closest in time.

X. Liu, T. Zhang, Y. Gu, I. L. Iong, Y. Xu, X. Song, S. Zhang, H. Lai, X. Liu, H. Zhao, J. Sun, X. Yang, Y. Yang, Z. Qi, S. Yao, X. Sun, S. Cheng, Q. Zheng, H. Yu, H. Zhang, W. Hong, M. Ding, L. Pan, X. Gu, A. Zeng, Z. Du, C. H. Song, Y. Su, Y. Dong, and J. Tang, “Visualagentbench: Towards large multimodal models as visual foundation agents,”

2024

Closest in time.

W. Wang, Y. Su, J. Huan, J. Liu, W. Chen, Y. Zhang, C.-Y. Li, K.-J. Chang, X. Xin, L. Shen

2024

Closest in time.

F. Bai, Y. Du, T. Huang, M. Q. H. Meng, and B. Zhao, “M3d: Advancing 3d medical image analysis with multi-modal large language models,”

2024

Closest in time.

J. Chen, R. Ouyang, A. Gao, S. Chen, G. H. Chen, X. Wang, R. Zhang, Z. Cai, K. Ji, G. Yu

2024

Closest in time.

P. Chen, J. Ye, G. Wang, Y. Li, Z. Deng, W. Li, T. Li, H. Duan, Z. Huang, Y. Su, B. Wang, S. Zhang, B. Fu, J. Cai, B. Zhuang, E. J. Seibel, J. He, and Y. Qiao, “Gmai-mmbench: A comprehensive multimodal evaluation benchmark towards general medical ai,”

2024

Closest in time.

J. Li, Y. Zhu, Z. Xu, J. Gu, M. Zhu, X. Liu, N. Liu, Y. Peng, F. Feng, and J. Tang, “Mmro: Are multimodal llms eligible as the brain for in-home robotics?”

2024

Closest in time.

P. Sermanet, T. Ding, J. Zhao, F. Xia, D. Dwibedi, K. Gopalakrishnan, C. Chan, G. Dulac-Arnold, S. Maddineni, N. J. Joshi

2024

Closest in time.

J. Lin, D. Huang, T. Zhao, D. Zhan, and C.-Y. Lin, “Designprobe: A graphic design benchmark for multimodal large language models,”

2024

Closest in time.

A. C. Doris, D. Grandi, R. Tomich, M. F. Alam, H. Cheong, and F. Ahmed, “Designqa: A multimodal benchmark for evaluating large language models’ understanding of engineering documentation,”

2024

Closest in time.

T. Yang, Y. Luo, Z. Qi, Y. Wu, Y. Shan, and C. W. Chen, “Posterllava: Constructing a unified multi-modal layout generator with llm,”

2024

Closest in time.

Y. Jin, M. Choi, G. Verma, J. Wang, and S. Kumar, “Mm-soc: Benchmarking multimodal large language models in social media platforms,”

2024

Closest in time.

X. Zhang, X. Shi, X. Lou, R. Qi, Y. Chen, J. Xu, and W. Han, “Transportationgames: Benchmarking transportation knowledge of (multimodal) large language models,”

2024

Closest in time.

T. Qian, J. Chen, L. Zhuo, Y. Jiao, and Y.-G. Jiang, “Nuscenes-qa: A multi-modal visual question answering benchmark for autonomous driving scenario,” in

2024

Closest in time.

D. Muhtar, Z. Li, F. Gu, X. Zhang, and P. Xiao, “Lhrs-bot: Empowering remote sensing with vgi-enhanced large multimodal language model,”

2024

Closest in time.

W. Street, J. O. Siy, G. Keeling, A. Baranes, B. Barnett, M. McKibben, T. Kanyere, A. Lentz, R. I. Dunbar

2024

Closest in time.

L. Zheng, W.-L. Chiang, Y. Sheng, S. Zhuang, Z. Wu, Y. Zhuang, Z. Lin, Z. Li, D. Li, E. Xing

2024

Closest in time.

L. Qu, H. Li, T. Wang, W. Wang, Y. Li, L. Nie, and T.-S. Chua, “Unified text-to-image generation and retrieval,”

2024

Closest in time.

Y. Qin, K. Song, Y. Hu, W. Yao, S. Cho, X. Wang, X. Wu, F. Liu, P. Liu, and D. Yu, “Infobench: Evaluating instruction following ability in large language models,”

2024

Closest in time.

Z. Bai, P. Wang, T. Xiao, T. He, Z. Han, Z. Zhang, and M. Z. Shou, “Hallucination of multimodal large language models: A survey,”

2024

Closest in time.

H. Liu, W. Xue, Y. Chen, D. Chen, X. Zhao, K. Wang, L. Hou, R. Li, and W. Peng, “A survey on hallucination in large vision-language models,”

2024

Closest in time.