ReLU to the Rescue: Improve Your On-Policy Actor-Critic with Positive Advantages

Understand

This paper proposes a step toward approximate Bayesian inference in on-policy actor-critic deep reinforcement learning.

• It is implemented through three changes to the Asynchronous Advantage Actor-Critic (A3C) algorithm: (1) applying a ReLU function to advantage estimates, (2) spectral normalization of actor-critic weights, and (3) incorporating \emph{dropout as a Bayesian approximation}.
• We prove under standard assumptions that restricting policy updates to positive advantages optimizes for value by maximizing a lower bound on the value function plus an additive term.
• We show that the additive term is bounded proportional to the Lipschitz constant of the value function, which offers theoretical grounding for spectral normalization of critic weights.