Resolving Language and Vision Ambiguities Together: Joint Segmentation & Prepositional Attachment Resolution in Captioned Scenes

Understand

We present an approach to simultaneously perform semantic segmentation and prepositional phrase attachment resolution for captioned images.

• Some ambiguities in language cannot be resolved without simultaneously reasoning about an associated image.
• If we consider the sentence "I shot an elephant in my pajamas", looking at language alone (and not using common sense), it is unclear if it is the person or the elephant wearing the pajamas or both.
• Our approach produces a diverse set of plausible hypotheses for both semantic segmentation and prepositional phrase attachment resolution that are then jointly reranked to select the most consistent pair.