Dr-LLaVA: Visual Instruction Tuning with Symbolic Clinical Grounding

Understand

Vision-Language Models (VLM) can support clinicians by analyzing medical images and engaging in natural language interactions to assist in diagnostic and treatment tasks.

• However, VLMs often exhibit "hallucinogenic" behavior, generating textual outputs not grounded in contextual multimodal information.
• This challenge is particularly pronounced in the medical domain, where we do not only require VLM outputs to be accurate in single interactions but also to be consistent with clinical reasoning and diagnostic pathways throughout multi-turn conversations.
• For this purpose, we propose a new alignment algorithm that uses symbolic representations of clinical reasoning to ground VLMs in medical knowledge.