A Comparative Analysis of Expected and Distributional Reinforcement Learning

Understand

Since their introduction a year ago, distributional approaches to reinforcement learning (distributional RL) have produced strong results relative to the standard approach which models expected values (expected RL).

• However, aside from convergence guarantees, there have been few theoretical results investigating the reasons behind the improvements distributional RL provides.
• In this paper we begin the investigation into this fundamental question by analyzing the differences in the tabular, linear approximation, and non-linear approximation settings.
• We prove that in many realizations of the tabular and linear approximation settings, distributional RL behaves exactly the same as expected RL.