PEGASUS: A Policy Search Method for Large MDPs and POMDPs

Understand

We propose a new approach to the problem of searching a space of policies for a Markov decision process (MDP) or a partially observable Markov decision process (POMDP), given a model.

• Our approach is based on the following observation: Any (PO)MDP can be transformed into an "equivalent" POMDP in which all state transitions (given the current state and action) are deterministic.
• This reduces the general problem of policy search to one in which we need only consider POMDPs with deterministic transitions.
• We give a natural way of estimating the value of all policies in these transformed POMDPs.