GeReA: Question-Aware Prompt Captions for Knowledge-based Visual Question Answering

Understand

Knowledge-based visual question answering (VQA) requires world knowledge beyond the image for accurate answer.

• Recently, instead of extra knowledge bases, a large language model (LLM) like GPT-3 is activated as an implicit knowledge engine to jointly acquire and reason the necessary knowledge for answering by converting images into textual information (e.g., captions and answer candidates).
• However, such conversion may introduce irrelevant information, which causes the LLM to misinterpret images and ignore visual details crucial for accurate knowledge.
• We argue that multimodal large language model (MLLM) is a better implicit knowledge engine than the LLM for its superior capability of visual understanding.