Learning Finite-State Controllers for Partially Observable Environments

Understand

Reactive (memoryless) policies are sufficient in completely observable Markov decision processes (MDPs), but some kind of memory is usually necessary for optimal control of a partially observable MDP.

• Policies with finite memory can be represented as finite-state automata.
• In this paper, we extend Baird and Moore's VAPS algorithm to the problem of learning general finite-state automata.
• Because it performs stochastic gradient descent, this algorithm can be shown to converge to a locally optimal finite-state controller.