Direct Language Model Alignment from Online AI Feedback

Understand

Direct alignment from preferences (DAP) methods, such as DPO, have recently emerged as efficient alternatives to reinforcement learning from human feedback (RLHF), that do not require a separate reward model.

• However, the preference datasets used in DAP methods are usually collected ahead of training and never updated, thus the feedback is purely offline.
• Moreover, responses in these datasets are often sampled from a language model distinct from the one being aligned, and since the model evolves over training, the alignment phase is inevitably off-policy.
• In this study, we posit that online feedback is key and improves DAP methods.