LMUnit: Fine-grained Evaluation with Natural Language Unit Tests

Understand

As language models become integral to critical workflows, assessing their behavior remains a fundamental challenge -- human evaluation is costly and noisy, while automated metrics provide only coarse, difficult-to-interpret signals.

• We introduce natural language unit tests, a paradigm that decomposes response quality into explicit, testable criteria, along with a unified scoring model, LMUnit, which combines multi-objective training across preferences, direct ratings, and natural language rationales.
• Through controlled human studies, we show this paradigm significantly improves inter-annotator agreement and enables more effective LLM development workflows.
• LMUnit achieves state-of-the-art performance on evaluation benchmarks (FLASK, BigGenBench) and competitive results on RewardBench.