Least Squares Policy Iteration with Instrumental Variables vs. Direct Policy Search: Comparison Against Optimal Benchmarks Using Energy Storage

Understand

This paper studies approximate policy iteration (API) methods which use least-squares Bellman error minimization for policy evaluation.

• We address several of its enhancements, namely, Bellman error minimization using instrumental variables, least-squares projected Bellman error minimization, and projected Bellman error minimization using instrumental variables.
• We prove that for a general discrete-time stochastic control problem, Bellman error minimization using instrumental variables is equivalent to both variants of projected Bellman error minimization.
• An alternative to these API methods is direct policy search based on knowledge gradient.