Likelihood-based Mitigation of Evaluation Bias in Large Language Models

Understand

Large Language Models (LLMs) are widely used to evaluate natural language generation tasks as automated metrics.

• However, the likelihood, a measure of LLM's plausibility for a sentence, can vary due to superficial differences in sentences, such as word order and sentence structure.
• It is therefore possible that there might be a likelihood bias if LLMs are used for evaluation: they might overrate sentences with higher likelihoods while underrating those with lower likelihoods.
• In this paper, we investigate the presence and impact of likelihood bias in LLM-based evaluators.