Understanding Forgetting in LLM Supervised Fine-Tuning and Preference Learning -- A Convex Optimization Perspective

Understand

The post-training of LLMs, which typically consists of the supervised fine-tuning (SFT) stage and the preference learning stage (RLHF or DPO), is crucial to effective and safe LLM applications.

• The widely adopted approach in post-training popular open-source LLMs is to sequentially perform SFT and RLHF/DPO.
• However, this is suboptimal in terms of SFT and RLHF/DPO trade-off: the LLM gradually forgets about the first stage's training when undergoing the second stage's training.
• This sequential paradigm persists largely due to its simplicity and modularity, which make it easier to implement and manage at scale despite its limitations.