Integrating Text and Image: Determining Multimodal Document Intent in Instagram Posts

Understand

Computing author intent from multimodal data like Instagram posts requires modeling a complex relationship between text and image.

• For example, a caption might evoke an ironic contrast with the image, so neither caption nor image is a mere transcript of the other.
• Instead they combine -- via what has been called meaning multiplication -- to create a new meaning that has a more complex relation to the literal meanings of text and image.
• Here we introduce a multimodal dataset of 1299 Instagram posts labeled for three orthogonal taxonomies: the authorial intent behind the image-caption pair, the contextual relationship between the literal meanings of the image and caption, and the semiotic relationship between the signified meanings of the image and caption.