Model Utility Law: Evaluating LLMs beyond Performance through Mechanism Interpretable Metric

Understand

Large Language Models (LLMs) have become indispensable across academia, industry, and daily applications, yet current evaluation methods struggle to keep pace with their rapid development.

• One core challenge of evaluation in the large language model (LLM) era is the generalization issue: how to infer a model's near-unbounded abilities from inevitably bounded benchmarks.
• We address this challenge by proposing Model Utilization Index (MUI), a mechanism interpretability enhanced metric that complements traditional performance scores.
• MUI quantifies the effort a model expends on a task, defined as the proportion of activated neurons or features during inference.