Can ChatGPT evaluate research quality?

Understand

Purpose: Assess whether ChatGPT 4.0 is accurate enough to perform research evaluations on journal articles to automate this time-consuming task.

• Design/methodology/approach: Test the extent to which ChatGPT-4 can assess the quality of journal articles using a case study of the published scoring guidelines of the UK Research Excellence Framework (REF) 2021 to create a research evaluation ChatGPT.
• This was applied to 51 of my own articles and compared against my own quality judgements.
• Findings: ChatGPT-4 can produce plausible document summaries and quality evaluation rationales that match the REF criteria.