Going Beneath the Surface: Evaluating Image Captioning for Grammaticality, Truthfulness and Diversity

Understand

Image captioning as a multimodal task has drawn much interest in recent years.

• However, evaluation for this task remains a challenging problem.
• Existing evaluation metrics focus on surface similarity between a candidate caption and a set of reference captions, and do not check the actual relation between a caption and the underlying visual content.
• We introduce a new diagnostic evaluation framework for the task of image captioning, with the goal of directly assessing models for grammaticality, truthfulness and diversity (GTD) of generated captions.