Online DPO: Online Direct Preference Optimization with Fast-Slow Chasing

Understand

Direct Preference Optimization (DPO) improves the alignment of large language models (LLMs) with human values by training directly on human preference datasets, eliminating the need for reward models.

• However, due to the presence of cross-domain human preferences, direct continual training can lead to catastrophic forgetting, limiting DPO's performance and efficiency.
• Inspired by intraspecific competition driving species evolution, we propose a Online Fast-Slow chasing DPO (OFS-DPO) for preference alignment, simulating competition through fast and slow chasing among models to facilitate rapid adaptation.
• Specifically, we first derive the regret upper bound for online learning, validating our motivation with a min-max optimization pattern.