Rethinking Learning Rate Tuning in the Era of Large Language Models

Understand

Large Language Models (LLMs) represent the recent success of deep learning in achieving remarkable human-like predictive performance.

• It has become a mainstream strategy to leverage fine-tuning to adapt LLMs for various real-world applications due to the prohibitive expenses associated with LLM training.
• The learning rate is one of the most important hyperparameters in LLM fine-tuning with direct impacts on both fine-tuning efficiency and fine-tuned LLM quality.
• Existing learning rate policies are primarily designed for training traditional deep neural networks (DNNs), which may not work well for LLM fine-tuning.