EMaQ: Expected-Max Q-Learning Operator for Simple Yet Effective Offline and Online RL

Understand

Off-policy reinforcement learning holds the promise of sample-efficient learning of decision-making policies by leveraging past experience.

• However, in the offline RL setting -- where a fixed collection of interactions are provided and no further interactions are allowed -- it has been shown that standard off-policy RL methods can significantly underperform.
• Recently proposed methods often aim to address this shortcoming by constraining learned policies to remain close to the given dataset of interactions.
• In this work, we closely investigate an important simplification of BCQ -- a prior approach for offline RL -- which removes a heuristic design choice and naturally restricts extracted policies to remain exactly within the support of a given behavior policy.