CLEME: Debiasing Multi-reference Evaluation for Grammatical Error Correction

Understand

Evaluating the performance of Grammatical Error Correction (GEC) systems is a challenging task due to its subjectivity.

• Designing an evaluation metric that is as objective as possible is crucial to the development of GEC task.
• However, mainstream evaluation metrics, i.e., reference-based metrics, introduce bias into the multi-reference evaluation by extracting edits without considering the presence of multiple references.
• To overcome this issue, we propose Chunk-LEvel Multi-reference Evaluation (CLEME), designed to evaluate GEC systems in the multi-reference evaluation setting.