Highway Reinforcement Learning

Understand

Learning from multi-step off-policy data collected by a set of policies is a core problem of reinforcement learning (RL).

• Approaches based on importance sampling (IS) often suffer from large variances due to products of IS ratios.
• Typical IS-free methods, such as $n$-step Q-learning, look ahead for $n$ time steps along the trajectory of actions (where $n$ is called the lookahead depth) and utilize off-policy data directly without any additional adjustment.
• They work well for proper choices of $n$.