COPR: Continual Human Preference Learning via Optimal Policy Regularization

Understand

Reinforcement Learning from Human Feedback (RLHF) is commonly utilized to improve the alignment of Large Language Models (LLMs) with human preferences.

• Given the evolving nature of human preferences, continual alignment becomes more crucial and practical in comparison to traditional static alignment.
• Nevertheless, making RLHF compatible with Continual Learning (CL) is challenging due to its complex process.
• Meanwhile, directly learning new human preferences may lead to Catastrophic Forgetting (CF) of historical preferences, resulting in helpless or harmful outputs.