Large-scale Hybrid Approach for Predicting User Satisfaction with Conversational Agents

Understand

Measuring user satisfaction level is a challenging task, and a critical component in developing large-scale conversational agent systems serving the needs of real users.

• An widely used approach to tackle this is to collect human annotation data and use them for evaluation or modeling.
• Human annotation based approaches are easier to control, but hard to scale.
• A novel alternative approach is to collect user's direct feedback via a feedback elicitation system embedded to the conversational agent system, and use the collected user feedback to train a machine-learned model for generalization.