Taylor Expansion of Discount Factors

Understand

In practical reinforcement learning (RL), the discount factor used for estimating value functions often differs from that used for defining the evaluation objective.

• In this work, we study the effect that this discrepancy of discount factors has during learning, and discover a family of objectives that interpolate value functions of two distinct discount factors.
• Our analysis suggests new ways for estimating value functions and performing policy optimization updates, which demonstrate empirical performance gains.
• This framework also leads to new insights on commonly-used deep RL heuristic modifications to policy optimization algorithms.