Grounding of Textual Phrases in Images by Reconstruction

Understand

Grounding (i.e.

• localizing) arbitrary, free-form textual phrases in visual content is a challenging problem with many applications for human-computer interaction and image-text reference resolution.
• Few datasets provide the ground truth spatial localization of phrases, thus it is desirable to learn from data with no or little grounding supervision.
• We propose a novel approach which learns grounding by reconstructing a given phrase using an attention mechanism, which can be either latent or optimized directly.