Banishing LLM Hallucinations Requires Rethinking Generalization

Understand

Despite their powerful chat, coding, and reasoning abilities, Large Language Models (LLMs) frequently hallucinate.

• Conventional wisdom suggests that hallucinations are a consequence of a balance between creativity and factuality, which can be mitigated, but not eliminated, by grounding the LLM in external knowledge sources.
• Through extensive systematic experiments, we show that these traditional approaches fail to explain why LLMs hallucinate in practice.
• Specifically, we show that LLMs augmented with a massive Mixture of Memory Experts (MoME) can easily memorize large datasets of random numbers.