The History and Risks of Reinforcement Learning and Human Feedback

Understand

Reinforcement learning from human feedback (RLHF) has emerged as a powerful technique to make large language models (LLMs) easier to use and more effective.

• A core piece of the RLHF process is the training and utilization of a model of human preferences that acts as a reward function for optimization.
• This approach, which operates at the intersection of many stakeholders and academic disciplines, remains poorly understood.
• RLHF reward models are often cited as being central to achieving performance, yet very few descriptors of capabilities, evaluations, training methods, or open-source models exist.