StoryDALL-E: Adapting Pretrained Text-to-Image Transformers for Story Continuation

Understand

Recent advances in text-to-image synthesis have led to large pretrained transformers with excellent capabilities to generate visualizations from a given text.

• However, these models are ill-suited for specialized tasks like story visualization, which requires an agent to produce a sequence of images given a corresponding sequence of captions, forming a narrative.
• Moreover, we find that the story visualization task fails to accommodate generalization to unseen plots and characters in new narratives.
• Hence, we first propose the task of story continuation, where the generated visual story is conditioned on a source image, allowing for better generalization to narratives with new characters.