Reward Centering

Understand

We show that discounted methods for solving continuing reinforcement learning problems can perform significantly better if they center their rewards by subtracting out the rewards' empirical average.

• The improvement is substantial at commonly used discount factors and increases further as the discount factor approaches one.
• In addition, we show that if a problem's rewards are shifted by a constant, then standard methods perform much worse, whereas methods with reward centering are unaffected.
• Estimating the average reward is straightforward in the on-policy setting; we propose a slightly more sophisticated method for the off-policy setting.