Evaluating the Factuality of Zero-shot Summarizers Across Varied Domains

Understand

Recent work has shown that large language models (LLMs) are capable of generating summaries zero-shot (i.e., without explicit supervision) that, under human assessment, are often comparable or even preferred to manually composed reference summaries.

• However, this prior work has focussed almost exclusively on evaluating news article summarization.
• How do zero-shot summarizers perform in other (potentially more specialized) domains? In this work we evaluate zero-shot generated summaries across specialized domains including biomedical articles, and legal bills (in addition to standard news benchmarks for reference).
• We focus especially on the factuality of outputs.