Is GPT-4 a reliable rater? Evaluating Consistency in GPT-4 Text Ratings

Understand

This study investigates the consistency of feedback ratings generated by OpenAI's GPT-4, a state-of-the-art artificial intelligence language model, across multiple iterations, time spans and stylistic variations.

• The model rated responses to tasks within the Higher Education (HE) subject domain of macroeconomics in terms of their content and style.
• Statistical analysis was conducted in order to learn more about the interrater reliability, consistency of the ratings across iterations and the correlation between ratings in terms of content and style.
• The results revealed a high interrater reliability with ICC scores ranging between 0.94 and 0.99 for different timespans, suggesting that GPT-4 is capable of generating consistent ratings across repetitions with a clear prompt.