Consistent Dropout for Policy Gradient Reinforcement Learning

Understand

Dropout has long been a staple of supervised learning, but is rarely used in reinforcement learning.

• We analyze why naive application of dropout is problematic for policy-gradient learning algorithms and introduce consistent dropout, a simple technique to address this instability.
• We demonstrate consistent dropout enables stable training with A2C and PPO in both continuous and discrete action environments across a wide range of dropout probabilities.
• Finally, we show that consistent dropout enables the online training of complex architectures such as GPT without needing to disable the model's native dropout.