Asymmetric REINFORCE for off-Policy Reinforcement Learning: Balancing positive and negative rewards

Understand

Reinforcement learning (RL) is increasingly used to align large language models (LLMs).

• Off-policy methods offer greater implementation simplicity and data efficiency than on-policy techniques, but often result in suboptimal performance.
• In this work, we study the intermediate range of algorithms between off-policy RL and supervised fine-tuning by analyzing a simple off-policy REINFORCE algorithm, where the advantage is defined as $A=r-V$, with $r$ a reward and $V$ some tunable baseline.
• Intuitively, lowering $V$ emphasizes high-reward samples, while raising it penalizes low-reward ones more heavily.