Average-Reward Off-Policy Policy Evaluation with Function Approximation

Understand

We consider off-policy policy evaluation with function approximation (FA) in average-reward MDPs, where the goal is to estimate both the reward rate and the differential value function.

• For this problem, bootstrapping is necessary and, along with off-policy learning and FA, results in the deadly triad (Sutton & Barto, 2018).
• To address the deadly triad, we propose two novel algorithms, reproducing the celebrated success of Gradient TD algorithms in the average-reward setting.
• In terms of estimating the differential value function, the algorithms are the first convergent off-policy linear function approximation algorithms.