Generalized Emphatic Temporal Difference Learning: Bias-Variance Analysis

Understand

We consider the off-policy evaluation problem in Markov decision processes with function approximation.

• We propose a generalization of the recently introduced \emph{emphatic temporal differences} (ETD) algorithm \citep{SuttonMW15}, which encompasses the original ETD($\lambda$), as well as several other off-policy evaluation algorithms as special cases.
• We call this framework \ETD, where our introduced parameter $\beta$ controls the decay rate of an importance-sampling term.
• We study conditions under which the projected fixed-point equation underlying \ETD\ involves a contraction operator, allowing us to present the first asymptotic error bounds (bias) for \ETD.