Attention Correctness in Neural Image Captioning

Understand

Attention mechanisms have recently been introduced in deep learning for various tasks in natural language processing and computer vision.

• But despite their popularity, the "correctness" of the implicitly-learned attention maps has only been assessed qualitatively by visualization of several examples.
• In this paper we focus on evaluating and improving the correctness of attention in neural image captioning models.
• Specifically, we propose a quantitative evaluation metric for the consistency between the generated attention maps and human annotations, using recently released datasets with alignment between regions in images and entities in captions.