Rethinking the Discount Factor in Reinforcement Learning: A Decision Theoretic Approach

Understand

Reinforcement learning (RL) agents have traditionally been tasked with maximizing the value function of a Markov decision process (MDP), either in continuous settings, with fixed discount factor $\gamma < 1$, or in episodic settings, with $\gamma = 1$.

• While this has proven effective for specific tasks with well-defined objectives (e.g., games), it has never been established that fixed discounting is suitable for general purpose use (e.g., as a model of human preferences).
• This paper characterizes rationality in sequential decision making using a set of seven axioms and arrives at a form of discounting that generalizes traditional fixed discounting.
• In particular, our framework admits a state-action dependent &#34;discount&#34; factor that is not constrained to be less than 1, so long as there is eventual long run discounting.