Teaching Large Language Models to Reason with Reinforcement Learning

Understand

Reinforcement Learning from Human Feedback (\textbf{RLHF}) has emerged as a dominant approach for aligning LLM outputs with human preferences.

• Inspired by the success of RLHF, we study the performance of multiple algorithms that learn from feedback (Expert Iteration, Proximal Policy Optimization (\textbf{PPO}), Return-Conditioned RL) on improving LLM reasoning capabilities.
• We investigate both sparse and dense rewards provided to the LLM both heuristically and via a learned reward model.
• We additionally start from multiple model sizes and initializations both with and without supervised fine-tuning (\textbf{SFT}) data.