ScaleCap: Inference-Time Scalable Image Captioning via Dual-Modality Debiasing

Understand

This paper presents ScaleCap, an inference-time scalable image captioning strategy that generates comprehensive and detailed image captions.

• The key challenges of high-quality image captioning lie in the inherent biases of LVLMs: multimodal bias resulting in imbalanced descriptive granularity, offering detailed accounts of some elements while merely skimming over others; linguistic bias leading to hallucinated descriptions of non-existent objects.
• To address these issues, we propose a scalable debiased captioning strategy, which continuously enriches and calibrates the caption with increased inference budget.
• Specifically, we propose two novel components: heuristic question answering and contrastive sentence rating.