How to Combine Tree-Search Methods in Reinforcement Learning

Understand

Finite-horizon lookahead policies are abundantly used in Reinforcement Learning and demonstrate impressive empirical success.

• Usually, the lookahead policies are implemented with specific planning methods such as Monte Carlo Tree Search (e.g.
• in AlphaZero).
• Referring to the planning problem as tree search, a reasonable practice in these implementations is to back up the value only at the leaves while the information obtained at the root is not leveraged other than for updating the policy.