Fine-tuning Language Models with Generative Adversarial Reward Modelling

Understand

Reinforcement Learning with Human Feedback (RLHF) has been demonstrated to significantly enhance the performance of large language models (LLMs) by aligning their outputs with desired human values through instruction tuning.

• However, RLHF is constrained by the expertise and productivity limitations of human evaluators.
• A response to this downside is to fall back to supervised fine-tuning (SFT) with additional carefully selected expert demonstrations.
• However, while this method has been proven to be effective, it invariably also leads to increased human-in-the-loop overhead.