LLaVA-VSD: Large Language-and-Vision Assistant for Visual Spatial Description

Understand

Visual Spatial Description (VSD) aims to generate texts that describe the spatial relationships between objects within images.

• Traditional visual spatial relationship classification (VSRC) methods typically output the spatial relationship between two objects in an image, often neglecting world knowledge and lacking general language capabilities.
• In this paper, we propose a Large Language-and-Vision Assistant for Visual Spatial Description, named LLaVA-VSD, which is designed for the classification, description, and open-ended description of visual spatial relationships.
• Specifically, the model first constructs a VSD instruction-following dataset using given figure-caption pairs for the three tasks.