On Optimistic versus Randomized Exploration in Reinforcement Learning

Understand

We discuss the relative merits of optimistic and randomized approaches to exploration in reinforcement learning.

• Optimistic approaches presented in the literature apply an optimistic boost to the value estimate at each state-action pair and select actions that are greedy with respect to the resulting optimistic value function.
• Randomized approaches sample from among statistically plausible value functions and select actions that are greedy with respect to the random sample.
• Prior computational experience suggests that randomized approaches can lead to far more statistically efficient learning.