Combining Language and Vision with a Multimodal Skip-gram Model

Understand

We extend the SKIP-GRAM model of Mikolov et al.

• (2013a) by taking visual information into account.
• Like SKIP-GRAM, our multimodal models (MMSKIP-GRAM) build vector-based word representations by learning to predict linguistic contexts in text corpora.
• However, for a restricted set of words, the models are also exposed to visual representations of the objects they denote (extracted from natural images), and must predict linguistic and visual features jointly.