HaluMem: Evaluating Hallucinations in Memory Systems of Agents

Understand

Memory systems are key components that enable AI systems such as LLMs and AI agents to achieve long-term learning and sustained interaction.

• However, during memory storage and retrieval, these systems frequently exhibit memory hallucinations, including fabrication, errors, conflicts, and omissions.
• Existing evaluations of memory hallucinations are primarily end-to-end question answering, which makes it difficult to localize the operational stage within the memory system where hallucinations arise.
• To address this, we introduce the Hallucination in Memory Benchmark (HaluMem), the first operation level hallucination evaluation benchmark tailored to memory systems.