Multi-step Reinforcement Learning: A Unifying Algorithm

Understand

Unifying seemingly disparate algorithmic ideas to produce better performing algorithms has been a longstanding goal in reinforcement learning.

• As a primary example, TD($\lambda$) elegantly unifies one-step TD prediction with Monte Carlo methods through the use of eligibility traces and the trace-decay parameter $\lambda$.
• Currently, there are a multitude of algorithms that can be used to perform TD control, including Sarsa, $Q$-learning, and Expected Sarsa.
• These methods are often studied in the one-step case, but they can be extended across multiple time steps to achieve better performance.