Combining Q-Learning and Search with Amortized Value Estimates

Understand

We introduce "Search with Amortized Value Estimates" (SAVE), an approach for combining model-free Q-learning with model-based Monte-Carlo Tree Search (MCTS).

• In SAVE, a learned prior over state-action values is used to guide MCTS, which estimates an improved set of state-action values.
• The new Q-estimates are then used in combination with real experience to update the prior.
• This effectively amortizes the value computation performed by MCTS, resulting in a cooperative relationship between model-free learning and model-based search.