Designing Precise and Robust Dialogue Response Evaluators

Understand

Automatic dialogue response evaluator has been proposed as an alternative to automated metrics and human evaluation.

• However, existing automatic evaluators achieve only moderate correlation with human judgement and they are not robust.
• In this work, we propose to build a reference-free evaluator and exploit the power of semi-supervised training and pretrained (masked) language models.
• Experimental results demonstrate that the proposed evaluator achieves a strong correlation (> 0.6) with human judgement and generalizes robustly to diverse responses and corpora.