Learning to Model Multimodal Semantic Alignment for Story Visualization

Understand

Story visualization aims to generate a sequence of images to narrate each sentence in a multi-sentence story, where the images should be realistic and keep global consistency across dynamic scenes and characters.

• Current works face the problem of semantic misalignment because of their fixed architecture and diversity of input modalities.
• To address this problem, we explore the semantic alignment between text and image representations by learning to match their semantic levels in the GAN-based generative model.
• More specifically, we introduce dynamic interactions according to learning to dynamically explore various semantic depths and fuse the different-modal information at a matched semantic level, which thus relieves the text-image semantic misalignment problem.