Performance Gap in Entity Knowledge Extraction Across Modalities in Vision Language Models

Understand

Vision-language models (VLMs) excel at extracting and reasoning about information from images.

• Yet, their capacity to leverage internal knowledge about specific entities remains underexplored.
• This work investigates the disparity in model performance when answering factual questions about an entity described in text versus depicted in an image.
• Our results reveal a significant accuracy drop - reaching 18% for some models - when the entity is presented visually instead of textually.