Hierarchically Structured Reinforcement Learning for Topically Coherent Visual Story Generation

Understand

We propose a hierarchically structured reinforcement learning approach to address the challenges of planning for generating coherent multi-sentence stories for the visual storytelling task.

• Within our framework, the task of generating a story given a sequence of images is divided across a two-level hierarchical decoder.
• The high-level decoder constructs a plan by generating a semantic concept (i.e., topic) for each image in sequence.
• The low-level decoder generates a sentence for each image using a semantic compositional network, which effectively grounds the sentence generation conditioned on the topic.