Asking and Answering Questions to Evaluate the Factual Consistency of Summaries

Understand

Practical applications of abstractive summarization models are limited by frequent factual inconsistencies with respect to their input.

• Existing automatic evaluation metrics for summarization are largely insensitive to such errors.
• We propose an automatic evaluation protocol called QAGS (pronounced "kags") that is designed to identify factual inconsistencies in a generated summary.
• QAGS is based on the intuition that if we ask questions about a summary and its source, we will receive similar answers if the summary is factually consistent with the source.