#Exploration: A Study of Count-Based Exploration for Deep Reinforcement Learning

Understand

Count-based exploration algorithms are known to perform near-optimally when used in conjunction with tabular reinforcement learning (RL) methods for solving small discrete Markov decision processes (MDPs).

• It is generally thought that count-based methods cannot be applied in high-dimensional state spaces, since most states will only occur once.
• Recent deep RL exploration strategies are able to deal with high-dimensional continuous state spaces through complex heuristics, often relying on optimism in the face of uncertainty or intrinsic motivation.
• In this work, we describe a surprising finding: a simple generalization of the classic count-based approach can reach near state-of-the-art performance on various high-dimensional and/or continuous deep RL benchmarks.