Sort Story: Sorting Jumbled Images and Captions into Stories

Understand

Temporal common sense has applications in AI tasks such as QA, multi-document summarization, and human-AI communication.

• We propose the task of sequencing -- given a jumbled set of aligned image-caption pairs that belong to a story, the task is to sort them such that the output sequence forms a coherent story.
• We present multiple approaches, via unary (position) and pairwise (order) predictions, and their ensemble-based combinations, achieving strong results on this task.
• We use both text-based and image-based features, which depict complementary improvements.