Lessons from the Trenches on Reproducible Evaluation of Language Models

Understand

Reliable evaluation of language models (LMs) remains an open challenge.

• Re- searchers and engineers face methodological issues such as the sensitivity of models to evaluation setup, difficulty of proper comparisons across methods, and the lack of reproducibility and transparency.
• Evaluation difficulties are exacer- bated by the fracturing and siloing of information about conventions and common practices.
• In this paper we draw on three years of experience in evaluating large lan- guage models (LMs) as developers of the popular Language Model Evaluation Harness (lm-eval) (Gao et al., 2023) framework to provide guidance and lessons for the field moving forward.