Fast Best-of-N Decoding via Speculative Rejection

Understand

The safe and effective deployment of Large Language Models (LLMs) involves a critical step called alignment, which ensures that the model's responses are in accordance with human preferences.

• Prevalent alignment techniques, such as DPO, PPO and their variants, align LLMs by changing the pre-trained model weights during a phase called post-training.
• While predominant, these post-training methods add substantial complexity before LLMs can be deployed.
• Inference-time alignment methods avoid the complex post-training step and instead bias the generation towards responses that are aligned with human preferences.