Cautious Optimizers: Improving Training with One Line of Code

Understand

AdamW has been the default optimizer for transformer pretraining.

• For many years, our community searched for faster and more stable optimizers with only constrained positive outcomes.
• In this work, we propose a \textbf{one-line modification in Pytorch} to any momentum-based optimizer, which we rename cautious optimizer, e.g.
• C-AdamW and C-Lion.