Learning Visually-Grounded Semantics from Contrastive Adversarial Samples

Understand

We study the problem of grounding distributional representations of texts on the visual domain, namely visual-semantic embeddings (VSE for short).

• Begin with an insightful adversarial attack on VSE embeddings, we show the limitation of current frameworks and image-text datasets (e.g., MS-COCO) both quantitatively and qualitatively.
• The large gap between the number of possible constitutions of real-world semantics and the size of parallel data, to a large extent, restricts the model to establish the link between textual semantics and visual concepts.
• We alleviate this problem by augmenting the MS-COCO image captioning datasets with textual contrastive adversarial samples.