OLMES: A Standard for Language Model Evaluations

Understand

Progress in AI is often demonstrated by new models claiming improved performance on tasks measuring model capabilities.

• Evaluating language models can be particularly challenging, as choices of how a model is evaluated on a task can lead to large changes in measured performance.
• There is no common standard setup, so different models are evaluated on the same tasks in different ways, leading to claims about which models perform best not being reproducible.
• We propose OLMES, a completely documented, practical, open standard for reproducible LLM evaluations.