Cost-of-Pass: An Economic Framework for Evaluating Language Models

Understand

Widespread adoption of AI systems hinges on their ability to generate economic value that outweighs their inference costs.

• Evaluating this tradeoff requires metrics accounting for both performance and costs.
• Building on production theory, we develop an economically grounded framework to evaluate language models' productivity by combining accuracy and inference cost.
• We formalize cost-of-pass: the expected monetary cost of generating a correct solution.