SuperHF: Supervised Iterative Learning from Human Feedback

Understand

While large language models demonstrate remarkable capabilities, they often present challenges in terms of safety, alignment with human values, and stability during training.

• Here, we focus on two prevalent methods used to align these models, Supervised Fine-Tuning (SFT) and Reinforcement Learning from Human Feedback (RLHF).
• SFT is simple and robust, powering a host of open-source models, while RLHF is a more sophisticated method used in top-tier models like ChatGPT but also suffers from instability and susceptibility to reward hacking.
• We propose a novel approach, Supervised Iterative Learning from Human Feedback (SuperHF), which seeks to leverage the strengths of both methods.