Benchmarking Multimodal Knowledge Conflict for Large Multimodal Models

Understand

Large Multimodal Models(LMMs) face notable challenges when encountering multimodal knowledge conflicts, particularly under retrieval-augmented generation(RAG) frameworks where the contextual information from external sources may contradict the model's internal parametric knowledge, leading to unreliable outputs.

• However, existing benchmarks fail to reflect such realistic conflict scenarios.
• Most focus solely on intra-memory conflicts, while context-memory and inter-context conflicts remain largely investigated.
• Furthermore, commonly used factual knowledge-based evaluations are often overlooked, and existing datasets lack a thorough investigation into conflict detection capabilities.