PRD: Peer Rank and Discussion Improve Large Language Model based Evaluations

Understand

Nowadays, the quality of responses generated by different modern large language models (LLMs) is hard to evaluate and compare automatically.

• Recent studies suggest and predominantly use LLMs for reference-free evaluation of open-ended question answering.
• More specifically, they use the recognized "strongest" LLM as the evaluator, which conducts pairwise comparisons of candidate models' answers and provides a ranking score.
• However, this intuitive method has multiple problems, such as bringing in self-enhancement (favoring its own answers) and positional bias.