Learning to Localize Objects Improves Spatial Reasoning in Visual-LLMs

Understand

Integration of Large Language Models (LLMs) into visual domain tasks, resulting in visual-LLMs (V-LLMs), has enabled exceptional performance in vision-language tasks, particularly for visual question answering (VQA).

• However, existing V-LLMs (e.g.
• BLIP-2, LLaVA) demonstrate weak spatial reasoning and localization awareness.
• Despite generating highly descriptive and elaborate textual answers, these models fail at simple tasks like distinguishing a left vs right location.