Do Automatic Factuality Metrics Measure Factuality? A Critical Evaluation

Understand

Modern LLMs can now produce highly readable abstractive summaries, to the point that traditional automated metrics for evaluating summary quality, such as ROUGE, have saturated.

• However, LLMs still sometimes introduce inaccuracies into summaries, i.e., information inconsistent with or unsupported by the corresponding source.
• Measuring the occurrence of these often subtle factual inconsistencies automatically has proved challenging.
• This in turn has motivated development of metrics intended to measure the factual consistency of generated summaries against sources.