RADDLE: An Evaluation Benchmark and Analysis Platform for Robust Task-oriented Dialog Systems

Understand

For task-oriented dialog systems to be maximally useful, it must be able to process conversations in a way that is (1) generalizable with a small number of training examples for new task domains, and (2) robust to user input in various styles, modalities or domains.

• In pursuit of these goals, we introduce the RADDLE benchmark, a collection of corpora and tools for evaluating the performance of models across a diverse set of domains.
• By including tasks with limited training data, RADDLE is designed to favor and encourage models with a strong generalization ability.
• RADDLE also includes a diagnostic checklist that facilitates detailed robustness analysis in aspects such as language variations, speech errors, unseen entities, and out-of-domain utterances.