Towards Best Experiment Design for Evaluating Dialogue System Output

Understand

To overcome the limitations of automated metrics (e.g.

• BLEU, METEOR) for evaluating dialogue systems, researchers typically use human judgments to provide convergent evidence.
• While it has been demonstrated that human judgments can suffer from the inconsistency of ratings, extant research has also found that the design of the evaluation task affects the consistency and quality of human judgments.
• We conduct a between-subjects study to understand the impact of four experiment conditions on human ratings of dialogue system output.