2020

Doubly Robust Off-Policy Value and Gradient Estimation for Deterministic Policies

Kallus, Nathan, Uehara, Masatoshi

Understand

Offline reinforcement learning, wherein one uses off-policy data logged by a fixed behavior policy to evaluate and learn new policies, is crucial in applications where experimentation is limited such as medicine.

We study the estimation of policy value and gradient of a deterministic policy from off-policy data when actions are continuous.
Targeting deterministic policies, for which action is a deterministic function of state, is crucial since optimal policies are always deterministic (up to ties).
In this setting, standard importance sampling and doubly robust estimators for policy value and gradient fail because the density ratio does not exist.

Built on

Orthogonal statistical learning
Original
Foster, D. J. and V. Syrgkanis (2019) · 1901
Earlier work this paper cites.
Double reinforcement learning for efficient off-policy evaluation in markov decision processes
Original
Kallus, N. and M. Uehara (2019a) · 1908
Earlier work this paper cites.
Efficiently breaking the curse of horizon: Double reinforcement learning in infinite-horizon processes
Original
Kallus, N. and M. Uehara (2019b) · 1909
Earlier work this paper cites.
From importance sampling to doubly robust policy gradient
Original
Huang, J. and N. Jiang (2019) · 1910
Earlier work this paper cites.
Eligibility Traces for Off-Policy Policy Evaluation
Precup, D., R. S. Sutton, and S. P. Singh (2000) · 2000
Earlier work this paper cites.
Statistically efficient off-policy policy gradients
Kallus, N. and M. Uehara (2020) · 2002
Earlier work this paper cites.
Optimal dynamic treatment regimes
Murphy, S. A. (2003) · 2003
Earlier work this paper cites.
Causal inference with general treatment regimes: Generalizing the propensity score
Imai, K. and D. A. van Dyk (2004) · 2004
Earlier work this paper cites.
The propensity score with continuous treatments
Hirano, K. and G. Imbens (2005) · 2005
Earlier work this paper cites.
Policy gradient methods for robotics
Peters, J. and S. Schaal (2006) · 2006
Earlier work this paper cites.
Chapter 76 large sample sieve estimation of semi-nonparametric models
Chen, X. (2007) · 2007
Earlier work this paper cites.
Nonparametric econometrics : theory and practice
Li, Q. and J. S. Racine (2007) · 2007
Earlier work this paper cites.
Learning near-optimal policies with bellman-residual minimization based fitted policy iteration and a single sample path
Antos, A., C. Szepesvári, and R. Munos (2008) · 2008
Earlier work this paper cites.
Finite-time bounds for fitted value iteration
Munos, R. and C. Szepesvári (2008) · 2008
Earlier work this paper cites.

Similar

Lecture notes on nonparametrics
Hansen, B. (2009) · 2009
Cited alongside, same era.
Mathematical Statistics : Asymptotic Minimax Theory
Korostelev, Alexander; Korosteleva, O. (2011) · 2011
Cited alongside, same era.
Deterministic policy gradient algorithms
Silver, D., G. Lever, N. Heess, T. Degris, D. Wierstra, and M. Riedmiller (2014) · 2014
Cited alongside, same era.
Uniformly semiparametric efficient estimation of treatment effects with a continuous treatment
Galvao, A. F. and L. Wang (2015) · 2015
Cited alongside, same era.
Doubly robust off-policy value evaluation for reinforcement learning
Jiang, N. and L. Li (2016) · 2016
Cited alongside, same era.
Data-efficient off-policy policy evaluation for reinforcement learning
Thomas, P. and E. Brunskill (2016) · 2016
Cited alongside, same era.
Efficient policy learning
Original
Athey, S. and S. Wager (2017) · 2017
Cited alongside, same era.
Data-adaptive smoothing for optimal-rate estimation of possibly non-regular parameters
Original
Bibaut, A. and M. J. van der Laan (2017) · 2017
Cited alongside, same era.
Non-parametric methods for doubly robust estimation of continuous treatment effects
Kennedy, E. H., Z. Ma, M. D. Mchugh, and D. S. Small (2017) · 2017
Cited alongside, same era.
Double/debiased machine learning for treatment and structural parameters
Chernozhukov, V., D. Chetverikov, M. Demirer, E. Duflo, C. Hansen, W. Newey, and J. Robins (2018) · 2018
Cited alongside, same era.
Locally robust semiparametric estimation
Chernozhukov, V., J. Escanciano, H. Ichimura, W. Newey, and J. Robins (2018) · 2018
Cited alongside, same era.
More robust doubly robust off-policy evaluation
Farajtabar, M., Y. Chow, and M. Ghavamzadeh (2018) · 2018
Cited alongside, same era.

Then

Policy evaluation and optimization with continuous treatments
Kallus, N. and A. Zhou (2018) · 2018
Later among the works it cites.
Partial mean processes with generated regressors: Continuous treatment effects and nonseparable models
Lee, Y.-Y. (2018) · 2018
Later among the works it cites.
Breaking the curse of horizon: Infinite-horizon off-policy estimation
Liu, Q., L. Li, Z. Tang, and D. Zhou (2018) · 2018
Later among the works it cites.
Representation balancing mdps for off-policy policy evaluation
Liu, Y., O. Gottesman, A. Raghu, M. Komorowski, A. A. Faisal, F. Doshi-Velez, and E. Brunskill (2018) · 2018
Later among the works it cites.
Matching on generalized propensity scores with continuous exposures
Original
Wu, X., F. Mealli, M.-A. Kioumourtzoglou, F. Dominici, and D. Braun (2018) · 2018
Later among the works it cites.
More efficient off-policy evaluation through regularized targeted learning
Bibaut, A., I. Malenica, N. Vlassis, and M. Van Der Laan (2019) · 2019
Later among the works it cites.
Double debiased machine learning nonparametric inference with continuous treatments
Colangelo, K. and Y.-Y. Lee (2019) · 2019
Later among the works it cites.
Guidelines for reinforcement learning in healthcare
Gottesman, O., F. Johansson, M. Komorowski, A. Faisal, D. Sontag, F. Doshi-Velez, and L. A. Celi (2019) · 2019
Later among the works it cites.
Causal Inference
Hernan, M. and J. Robins (2019) · 2019
Later among the works it cites.
Batch policy learning under constraints
Le, H., C. Voloshin, and Y. Yue (2019) · 2019
Later among the works it cites.
Non-separable models with high-dimensional data
Su, L., T. Ura, and Y. Zhang (2019) · 2019
Later among the works it cites.
Towards optimal off-policy evaluation for reinforcement learning with marginalized importance sampling
Xie, T., Y. Ma, and Y.-X. Wang (2019) · 2019
Later among the works it cites.
Asymptotically efficient off-policy evaluation for tabular reinforcement learning
Original
Yin, M. and Y.-X. Wang (2020) · 2020
Closest in time.
Off-policy deep reinforcement learning without exploration
Fujimoto, S., D. Meger, and D. Precup (2019) · 2062
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…