Context Does Matter: Implications for Crowdsourced Evaluation Labels in Task-Oriented Dialogue Systems

Understand

Crowdsourced labels play a crucial role in evaluating task-oriented dialogue systems (TDSs).

• Obtaining high-quality and consistent ground-truth labels from annotators presents challenges.
• When evaluating a TDS, annotators must fully comprehend the dialogue before providing judgments.
• Previous studies suggest using only a portion of the dialogue context in the annotation process.