Sycophancy in Vision-Language Models: A Systematic Analysis and an Inference-Time Mitigation Framework

Y. Liu, M. Ott, N. Goyal, J. Du, M. Joshi, D. Chen, O. Levy, M. Lewis, L. Zettlemoyer, V. Stoyanov, Roberta: A robustly optimized bert pretraining approach, arXiv preprint arXiv:1907.11692 (2019)

Original

2019

Earlier work this paper cites.

A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, et al., Learning transferable visual models from natural language supervision, in: International conference on machine learning, PMLR, 2021, pp. 8748–8763

2021

Earlier work this paper cites.

J. Ho, T. Salimans, Classifier-free diffusion guidance, in: NeurIPS 2021 Workshop on Deep Generative Models and Downstream Applications, 2021

2021

Earlier work this paper cites.

P. Lu, S. Mishra, T. Xia, L. Qiu, K.-W. Chang, S.-C. Zhu, O. Tafjord, P. Clark, A. Kalyan, Learn to explain: Multimodal reasoning via thought chains for science question answering, Advances in Neural Information Processing Systems 35 (2022) 2507–2521

2022

Earlier work this paper cites.

K. Han, Y. Wang, H. Chen, X. Chen, J. Guo, Z. Liu, Y. Tang, A. Xiao, C. Xu, Y. Xu, et al., A survey on vision transformer, IEEE transactions on pattern analysis and machine intelligence 45 (1) (2022) 87–110

2022

Earlier work this paper cites.

J. Wei, X. Wang, D. Schuurmans, M. Bosma, F. Xia, E. Chi, Q. V. Le, D. Zhou, et al., Chain-of-thought prompting elicits reasoning in large language models, Advances in neural information processing systems 35 (2022) 24824–24837

2022

Earlier work this paper cites.

E. Perez, S. Ringer, K. Lukosiute, K. Nguyen, E. Chen, S. Heiner, C. Pettit, C. Olsson, S. Kundu, S. Kadavath, et al., Discovering language model behaviors with model-written evaluations, in: Findings of the Association for Computational Linguistics: ACL 2023, 2023, pp. 13387–13434

2023

Earlier work this paper cites.

C. Cui, Y. Zhou, X. Yang, S. Wu, L. Zhang, J. Zou, H. Yao, Holistic analysis of hallucination in gpt-4v (ision): Bias and interference challenges, arXiv preprint arXiv:2311.03287 (2023)

Original

2023

Earlier work this paper cites.

Y. Li, Y. Du, K. Zhou, J. Wang, W. X. Zhao, J.-R. Wen, Evaluating object hallucination in large vision-language models, in: Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, 2023, pp. 292–305

2023

Earlier work this paper cites.

J. Wang, Y. Wang, G. Xu, J. Zhang, Y. Gu, H. Jia, M. Yan, J. Zhang, J. Sang, An llm-free multi-dimensional benchmark for mllms hallucination evaluation, arXiv preprint arXiv:2311.07397 (2023)

Original

2023

Earlier work this paper cites.

J. Bai, S. Bai, S. Yang, S. Wang, S. Tan, P. Wang, J. Lin, C. Zhou, J. Zhou, Qwen-vl: A frontier large vision-language model with versatile abilities, arXiv preprint arXiv:2308.12966 (2023)

Original

2023

Earlier work this paper cites.

X. L. Li, A. Holtzman, D. Fried, P. Liang, J. Eisner, T. B. Hashimoto, L. Zettlemoyer, M. Lewis, Contrastive decoding: Open-ended text generation as optimization, in: Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 2023, pp. 12286–12312

2023

Earlier work this paper cites.

C. Fu, P. Chen, Y. Shen, Y. Qin, M. Zhang, X. Lin, J. Yang, X. Zheng, K. Li, X. Sun, et al., Mme: A comprehensive evaluation benchmark for multimodal large language models, arXiv preprint arXiv:2306.13394 (2023)

Original

2023

Earlier work this paper cites.

W. Kwon, Z. Li, S. Zhuang, Y. Sheng, L. Zheng, C. H. Yu, J. E. Gonzalez, H. Zhang, I. Stoica, Efficient memory management for large language model serving with pagedattention, in: Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles, 2023

2023

Earlier work this paper cites.

doi:https://doi.org/10.1016/j.neucom.2024.127530

Y. Zhang, C. Zhang, Y. Tang, Z. He, Cross-modal concept learning and inference for vision-language models , Neurocomputing 583 (2024) 127530 · 2024

Cited alongside, same era.

doi:https://doi.org/10.1016/j.neucom.2024.129131

J. Rodriguez-Juan, D. Ortiz-Perez, J. Garcia-Rodriguez, D. Tomás, G. J.Nalepa, Integrating advanced vision-language models for context recognition in risks assessment , Neurocomputing 618 (2025) 129131 · 2024

Cited alongside, same era.

D. Zhang, Y. Yu, J. Dong, C. Li, D. Su, C. Chu, D. Yu, Mm-llms: Recent advances in multimodal large language models, in: Findings of the Association for Computational Linguistics ACL 2024, 2024, pp. 12401–12430

2024

Cited alongside, same era.

C. Cui, Y. Ma, X. Cao, W. Ye, Y. Zhou, K. Liang, J. Chen, J. Lu, Z. Yang, K.-D. Liao, et al., A survey on multimodal large language models for autonomous driving, in: Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, 2024, pp. 958–979

2024

Cited alongside, same era.

X. Liu, J. Wu, W. Yang, X. Zhou, T. Zhang, Multi-modal attribute prompting for vision-language models, IEEE Transactions on Circuits and Systems for Video Technology (2024)

2024

Closest in time.

M. Wu, J. Ji, O. Huang, J. Li, Y. Wu, X. Sun, R. Ji, Evaluating and analyzing relationship hallucinations in large vision-language models, in: Forty-first International Conference on Machine Learning, 2024

2024

Closest in time.

T. Guan, F. Liu, X. Wu, R. Xian, Z. Li, X. Liu, X. Wang, L. Chen, F. Huang, Y. Yacoob, et al., Hallusionbench: an advanced diagnostic suite for entangled language hallucination and visual illusion in large vision-language models, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024, pp. 14375–14385

2024

Closest in time.

Q. Huang, X. Dong, P. Zhang, B. Wang, C. He, J. Wang, D. Lin, W. Zhang, N. Yu, Opera: Alleviating hallucination in multi-modal large language models via over-trust penalty and retrospection-allocation, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024, pp. 13418–13427