Neural Language Modeling with Visual Features

Understand

Multimodal language models attempt to incorporate non-linguistic features for the language modeling task.

• In this work, we extend a standard recurrent neural network (RNN) language model with features derived from videos.
• We train our models on data that is two orders-of-magnitude bigger than datasets used in prior work.
• We perform a thorough exploration of model architectures for combining visual and text features.