Straight to Zero: Why Linearly Decaying the Learning Rate to Zero Works Best for LLMs

Understand

LLMs are commonly trained with a learning rate (LR) warmup, followed by cosine decay to 10% of the maximum (10x decay).

• In a large-scale empirical study, we show that under an optimal peak LR, a simple linear decay-to-zero (D2Z) schedule consistently outperforms other schedules when training at compute-optimal dataset sizes.
• D2Z is superior across a range of model sizes, batch sizes, datasets, and vocabularies.
• Benefits increase as dataset size increases.