Open-ended VQA benchmarking of Vision-Language models by exploiting Classification datasets and their semantic hierarchy

Understand

The evaluation of text-generative vision-language models is a challenging yet crucial endeavor.

• By addressing the limitations of existing Visual Question Answering (VQA) benchmarks and proposing innovative evaluation methodologies, our research seeks to advance our understanding of these models' capabilities.
• We propose a novel VQA benchmark based on well-known visual classification datasets which allows a granular evaluation of text-generative vision-language models and their comparison with discriminative vision-language models.
• To improve the assessment of coarse answers on fine-grained classification tasks, we suggest using the semantic hierarchy of the label space to ask automatically generated follow-up questions about the ground-truth category.