On Entropy Control in LLM-RL Algorithms

Understand

For RL algorithms, appropriate entropy control is crucial to their effectiveness.

• To control the policy entropy, a commonly used method is entropy regularization, which is adopted in various popular RL algorithms including PPO, SAC and A3C.
• Although entropy regularization proves effective in robotic and games RL conventionally, studies found that it gives weak to no gains in LLM-RL training.
• In this work, we study the issues of entropy bonus in LLM-RL setting.