Reward Tweaking: Maximizing the Total Reward While Planning for Short Horizons

Understand

In reinforcement learning, the discount factor $\gamma$ controls the agent's effective planning horizon.

• Traditionally, this parameter was considered part of the MDP; however, as deep reinforcement learning algorithms tend to become unstable when the effective planning horizon is long, recent works refer to $\gamma$ as a hyper-parameter -- thus changing the underlying MDP and potentially leading the agent towards sub-optimal behavior on the original task.
• In this work, we introduce \emph{reward tweaking}.
• Reward tweaking learns a surrogate reward function $\tilde r$ for the discounted setting that induces optimal behavior on the original finite-horizon total reward task.