Bootstrapping Language Models with DPO Implicit Rewards

Understand

Human alignment in large language models (LLMs) is an active area of research.

• A recent groundbreaking work, direct preference optimization (DPO), has greatly simplified the process from past work in reinforcement learning from human feedback (RLHF) by bypassing the reward learning stage in RLHF.
• DPO, after training, provides an implicit reward model.
• In this work, we make a novel observation that this implicit reward model can by itself be used in a bootstrapping fashion to further align the LLM.