Sample-Efficient Alignment for LLMs

Understand

We study methods for efficiently aligning large language models (LLMs) with human preferences given budgeted online feedback.

• We first formulate the LLM alignment problem in the frame of contextual dueling bandits.
• This formulation, subsuming recent paradigms such as online RLHF and online DPO, inherently quests for sample-efficient algorithms that incorporate online active exploration.
• Leveraging insights from bandit theory, we introduce a unified algorithm based on Thompson sampling and highlight its applications in two distinct LLM alignment scenarios.