GenCeption: Evaluate Vision LLMs with Unlabeled Unimodal Data

Understand

Multimodal Large Language Models (MLLMs) are typically assessed using expensive annotated multimodal benchmarks, which often lag behind the rapidly evolving demands of MLLM evaluation.

• This paper outlines and validates GenCeption, a novel, annotation-free evaluation method that requires only unimodal data to measure inter-modality semantic coherence and inversely assesses MLLMs' tendency to hallucinate.
• This approach eliminates the need for costly data annotation, minimizes the risk of training data contamination, is expected to result in slower benchmark saturation, and avoids the illusion of emerging abilities.
• Inspired by the DrawCeption game, GenCeption begins with a non-textual sample and proceeds through iterative description and generation steps.