Unsupervised Visual Sense Disambiguation for Verbs using Multimodal Embeddings

Understand

We introduce a new task, visual sense disambiguation for verbs: given an image and a verb, assign the correct sense of the verb, i.e., the one that describes the action depicted in the image.

• Just as textual word sense disambiguation is useful for a wide range of NLP tasks, visual sense disambiguation can be useful for multimodal tasks such as image retrieval, image description, and text illustration.
• We introduce VerSe, a new dataset that augments existing multimodal datasets (COCO and TUHOI) with sense labels.
• We propose an unsupervised algorithm based on Lesk which performs visual sense disambiguation using textual, visual, or multimodal embeddings.