Single-Agent Optimization Through Policy Iteration Using Monte-Carlo Tree Search

Understand

The combination of Monte-Carlo Tree Search (MCTS) and deep reinforcement learning is state-of-the-art in two-player perfect-information games.

• In this paper, we describe a search algorithm that uses a variant of MCTS which we enhanced by 1) a novel action value normalization mechanism for games with potentially unbounded rewards (which is the case in many optimization problems), 2) defining a virtual loss function that enables effective search parallelization, and 3) a policy network, trained by generations of self-play, to guide the search.
• We gauge the effectiveness of our method in "SameGame"---a popular single-player test domain.
• Our experimental results indicate that our method outperforms baseline algorithms on several board sizes.