On the Statistical Benefits of Temporal Difference Learning

Understand

Given a dataset on actions and resulting long-term rewards, a direct estimation approach fits value functions that minimize prediction error on the training data.

• Temporal difference learning (TD) methods instead fit value functions by minimizing the degree of temporal inconsistency between estimates made at successive time-steps.
• Focusing on finite state Markov chains, we provide a crisp asymptotic theory of the statistical advantages of this approach.
• First, we show that an intuitive inverse trajectory pooling coefficient completely characterizes the percent reduction in mean-squared error of value estimates.