Spot The Bot: A Robust and Efficient Framework for the Evaluation of Conversational Dialogue Systems

Understand

The lack of time-efficient and reliable evaluation methods hamper the development of conversational dialogue systems (chatbots).

• Evaluations requiring humans to converse with chatbots are time and cost-intensive, put high cognitive demands on the human judges, and yield low-quality results.
• In this work, we introduce \emph{Spot The Bot}, a cost-efficient and robust evaluation framework that replaces human-bot conversations with conversations between bots.
• Human judges then only annotate for each entity in a conversation whether they think it is human or not (assuming there are humans participants in these conversations).