LLMs on the Line: Data Determines Loss-to-Loss Scaling Laws

Understand

Scaling laws guide the development of large language models (LLMs) by offering estimates for the optimal balance of model size, tokens, and compute.

• More recently, loss-to-loss scaling laws that relate losses across pretraining datasets and downstream tasks have emerged as a powerful tool for understanding and improving LLM performance and generalization.
• In this work, we investigate which factors most strongly influence loss-to-loss scaling.
• Our experiments reveal that the pretraining data determines the scaling trend.