KG-FPQ: Evaluating Factuality Hallucination in LLMs with Knowledge Graph-based False Premise Questions

Understand

Recent studies have demonstrated that large language models (LLMs) are susceptible to being misled by false premise questions (FPQs), leading to errors in factual knowledge, know as factuality hallucination.

• Existing benchmarks that assess this vulnerability primarily rely on manual construction, resulting in limited scale and lack of scalability.
• In this work, we introduce an automated, scalable pipeline to create FPQs based on knowledge graphs (KGs).
• The first step is modifying true triplets extracted from KGs to create false premises.