Measuring the Measurers: Quality Evaluation of Hallucination Benchmarks for Large Vision-Language Models

Understand

Despite the outstanding performance in multimodal tasks, Large Vision-Language Models (LVLMs) have been plagued by the issue of hallucination, i.e., generating content that is inconsistent with the corresponding visual inputs.

• While previous works have proposed various benchmarks to evaluate this issue, the quality of these evaluations remains unverified.
• We observe that some of these benchmarks may produce inconsistent evaluation results across repeated tests or fail to align with human evaluation.
• To address this, we propose a Hallucination benchmark Quality Measurement framework (HQM), which leverages specific indicators to assess both reliability and validity.