Distributionally Robust Reinforcement Learning with Human Feedback

Understand

Reinforcement learning from human feedback (RLHF) has evolved to be one of the main methods for fine-tuning large language models (LLMs).

• However, existing RLHF methods are non-robust, and their performance deteriorates if the downstream task differs significantly from the preference dataset used in fine-tuning.
• In order to mitigate this problem, we introduce a distributionally robust RLHF for fine-tuning LLMs.
• In particular, our goal is to ensure that a fine-tuned model retains its performance even when the distribution of prompts significantly differs from the distribution encountered during fine-tuning.