A Technical Survey of Reinforcement Learning Techniques for Large Language Models

Understand

This survey offers a comprehensive foundation on the integration of RL with language models, highlighting prominent algorithms such as Proximal Policy Optimization (PPO), Q-Learning, and Actor-Critic methods.

• Additionally, it provides an extensive technical overview of RL techniques specifically tailored for LLMs, including foundational methods like Reinforcement Learning from Human Feedback (RLHF) and AI Feedback (RLAIF), as well as advanced strategies such as Direct Preference Optimization (DPO) and Group Relative Policy Optimization (GRPO).
• We systematically analyze their applications across domains, i.e., from code generation to tool-augmented reasoning.
• Crucially, we move beyond descriptive categorization to provide a rigorous algorithmic analysis of failure modes, mathematically framing the structural bottlenecks and stability trade-offs inherent in policy optimization.