Beyond Chinchilla-Optimal: Accounting for Inference in Language Model Scaling Laws

Understand

Large language model (LLM) scaling laws are empirical formulas that estimate changes in model quality as a result of increasing parameter count and training data.

• However, these formulas, including the popular Deepmind Chinchilla scaling laws, neglect to include the cost of inference.
• We modify the Chinchilla scaling laws to calculate the optimal LLM parameter count and pre-training data size to train and deploy a model of a given quality and inference demand.
• We conduct our analysis both in terms of a compute budget and real-world costs and find that LLM researchers expecting reasonably large inference demand (~1B requests) should train models smaller and longer than Chinchilla-optimal.