Leveraging LLMs for Dialogue Quality Measurement

Understand

In task-oriented conversational AI evaluation, unsupervised methods poorly correlate with human judgments, and supervised approaches lack generalization.

• Recent advances in large language models (LLMs) show robust zeroshot and few-shot capabilities across NLP tasks.
• This paper explores using LLMs for automated dialogue quality evaluation, experimenting with various configurations on public and proprietary datasets.
• Manipulating factors such as model size, in-context examples, and selection techniques, we examine "chain-of-thought" (CoT) reasoning and label extraction procedures.