Multimodal Foundation Models Exploit Text to Make Medical Image Predictions

Understand

Multimodal foundation models have shown compelling but conflicting performance in medical image interpretation.

• However, the mechanisms by which these models integrate and prioritize different data modalities, including images and text, remain poorly understood.
• Here, using a diverse collection of 1014 multimodal medical cases, we evaluate the unimodal and multimodal image interpretation abilities of proprietary (GPT-4, Gemini Pro 1.0) and open-source (Llama-3.2-90B, LLaVA-Med-v1.5) multimodal foundational models with and without the use of text descriptions.
• Across all models, image predictions were largely driven by exploiting text, with accuracy increasing monotonically with the amount of informative text.