Trust Me, I'm Wrong: LLMs Hallucinate with Certainty Despite Knowing the Answer

Understand

Prior work on large language model (LLM) hallucinations has associated them with model uncertainty or inaccurate knowledge.

• In this work, we define and investigate a distinct type of hallucination, where a model can consistently answer a question correctly, but a seemingly trivial perturbation, which can happen in real-world settings, causes it to produce a hallucinated response with high certainty.
• This phenomenon, which we dub CHOKE (Certain Hallucinations Overriding Known Evidence), is particularly concerning in high-stakes domains such as medicine or law, where model certainty is often used as a proxy for reliability.
• We show that CHOKE examples are consistent across prompts, occur in different models and datasets, and are fundamentally distinct from other hallucinations.