Learning language through pictures

Understand

We propose Imaginet, a model of learning visually grounded representations of language from coupled textual and visual input.

• The model consists of two Gated Recurrent Unit networks with shared word embeddings, and uses a multi-task objective by receiving a textual description of a scene and trying to concurrently predict its visual representation and the next word in the sentence.
• Mimicking an important aspect of human language learning, it acquires meaning representations for individual words from descriptions of visual scenes.
• Moreover, it learns to effectively use sequential structure in semantic interpretation of multi-word phrases.