Reference-Guided Verdict: LLMs-as-Judges in Automatic Evaluation of Free-Form QA

Understand

The emergence of Large Language Models (LLMs) as chat assistants capable of generating human-like conversations has amplified the need for robust evaluation methods, particularly for open-ended tasks.

• Conventional metrics such as EM and F1, while useful, are inadequate for capturing the full semantics and contextual depth of such generative outputs.
• We propose a reference-guided verdict method that automates the evaluation process by leveraging multiple LLMs as judges.
• Through experiments on free-form question-answering tasks, we demonstrate that combining multiple models improves the reliability and accuracy of evaluations, especially in tasks where a single model may struggle.