Text-guided Attention Model for Image Captioning

Understand

Visual attention plays an important role to understand images and demonstrates its effectiveness in generating natural language descriptions of images.

• On the other hand, recent studies show that language associated with an image can steer visual attention in the scene during our cognitive process.
• Inspired by this, we introduce a text-guided attention model for image captioning, which learns to drive visual attention using associated captions.
• For this model, we propose an exemplar-based learning approach that retrieves from training data associated captions with each image, and use them to learn attention on visual features.