Reference-based Metrics Disprove Themselves in Question Generation

Understand

Reference-based metrics such as BLEU and BERTScore are widely used to evaluate question generation (QG).

• In this study, on QG benchmarks such as SQuAD and HotpotQA, we find that using human-written references cannot guarantee the effectiveness of the reference-based metrics.
• Most QG benchmarks have only one reference; we replicate the annotation process and collect another reference.
• A good metric is expected to grade a human-validated question no worse than generated questions.