Re-Examining System-Level Correlations of Automatic Summarization Evaluation Metrics

Understand

How reliably an automatic summarization evaluation metric replicates human judgments of summary quality is quantified by system-level correlations.

• We identify two ways in which the definition of the system-level correlation is inconsistent with how metrics are used to evaluate systems in practice and propose changes to rectify this disconnect.
• First, we calculate the system score for an automatic metric using the full test set instead of the subset of summaries judged by humans, which is currently standard practice.
• We demonstrate how this small change leads to more precise estimates of system-level correlations.