INST-IT: Boosting Instance Understanding via Explicit Visual Prompt Instruction Tuning

Understand

Large Multimodal Models (LMMs) have made significant breakthroughs with the advancement of instruction tuning.

• However, while existing models can understand images and videos at a holistic level, they still struggle with instance-level understanding that requires a more fine-grained comprehension and alignment.
• Instance-level understanding is crucial for LMMs, as it focuses on the specific elements that we are most interested in.
• Excitingly, existing works find that the SOTA LMMs exhibit strong instance understanding capabilities when provided with explicit visual cues.