Reinforcement Learning Finetunes Small Subnetworks in Large Language Models

Understand

Reinforcement learning (RL) yields substantial improvements in large language models (LLMs) downstream task performance and alignment with human values.

• Surprisingly, such large gains result from updating only a small subnetwork comprising just 5 percent to 30 percent of the parameters, with the rest effectively unchanged.
• We refer to this phenomenon as parameter update sparsity induced by RL.
• It is observed across all 7 widely used RL algorithms (e.g., PPO, GRPO, DPO) and all 10 LLMs from different families in our experiments.