Is ChatGPT a Good NLG Evaluator? A Preliminary Study

Understand

Recently, the emergence of ChatGPT has attracted wide attention from the computational linguistics community.

• Many prior studies have shown that ChatGPT achieves remarkable performance on various NLP tasks in terms of automatic evaluation metrics.
• However, the ability of ChatGPT to serve as an evaluation metric is still underexplored.
• Considering assessing the quality of natural language generation (NLG) models is an arduous task and NLG metrics notoriously show their poor correlation with human judgments, we wonder whether ChatGPT is a good NLG evaluation metric.