Predictive Engagement: An Efficient Metric For Automatic Evaluation of Open-Domain Dialogue Systems

Understand

User engagement is a critical metric for evaluating the quality of open-domain dialogue systems.

• Prior work has focused on conversation-level engagement by using heuristically constructed features such as the number of turns and the total time of the conversation.
• In this paper, we investigate the possibility and efficacy of estimating utterance-level engagement and define a novel metric, {\em predictive engagement}, for automatic evaluation of open-domain dialogue systems.
• Our experiments demonstrate that (1) human annotators have high agreement on assessing utterance-level engagement scores; (2) conversation-level engagement scores can be predicted from properly aggregated utterance-level engagement scores.