Increasing the Action Gap: New Operators for Reinforcement Learning

Understand

This paper introduces new optimality-preserving operators on Q-functions.

• We first describe an operator for tabular representations, the consistent Bellman operator, which incorporates a notion of local policy consistency.
• We show that this local consistency leads to an increase in the action gap at each state; increasing this gap, we argue, mitigates the undesirable effects of approximation and estimation errors on the induced greedy policies.
• This operator can also be applied to discretized continuous space and time problems, and we provide empirical results evidencing superior performance in this context.