Reinforcement Learning with Exogenous States and Rewards

Understand

Exogenous state variables and rewards can slow reinforcement learning by injecting uncontrolled variation into the reward signal.

• This paper formalizes exogenous state variables and rewards and shows that if the reward function decomposes additively into endogenous and exogenous components, the MDP can be decomposed into an exogenous Markov Reward Process (based on the exogenous reward) and an endogenous Markov Decision Process (optimizing the endogenous reward).
• Any optimal policy for the endogenous MDP is also an optimal policy for the original MDP, but because the endogenous reward typically has reduced variance, the endogenous MDP is easier to solve.
• We study settings where the decomposition of the state space into exogenous and endogenous state spaces is not given but must be discovered.