Graph-Based Captioning: Enhancing Visual Descriptions by Interconnecting Region Captions

Understand

Humans describe complex scenes with compositionality, using simple text descriptions enriched with links and relationships.

• While vision-language research has aimed to develop models with compositional understanding capabilities, this is not reflected yet in existing datasets which, for the most part, still use plain text to describe images.
• In this work, we propose a new annotation strategy, graph-based captioning (GBC) that describes an image using a labeled graph structure, with nodes of various types.
• The nodes in GBC are created through a two-stage process: first, identifying and describing entity nodes; second, linking these nodes by highlighting \textit{compositions} and \textit{relations} among them.