2019

Learning When-to-Treat Policies

Nie, Xinkun, Brunskill, Emma, Wager, Stefan

Understand

Many applied decision-making problems have a dynamic component: The policymaker needs not only to choose whom to treat, but also when to start which treatment.

For example, a medical doctor may choose between postponing treatment (watchful waiting) and prescribing one of several available treatments during the many visits from a patient.
We develop an "advantage doubly robust" estimator for learning such dynamic treatment rules using observational data under the assumption of sequential ignorability.
We prove welfare regret bounds that generalize results for doubly robust learning in the single-step setting, and show promising empirical performance in several different contexts.

Built on

Sur les applications de la théorie des probabilités aux experiences agricoles: Essai des principes
J. Neyman · 1923
Earlier work this paper cites.
Estimating causal effects of treatments in randomized and nonrandomized studies
D. B. Rubin · 1974
Earlier work this paper cites.
On optimal stopping and free boundary problems
P. Van Moerbeke · 1976
Earlier work this paper cites.
The central role of the propensity score in observational studies for causal effects
P. R. Rosenbaum and D. B. Rubin · 1983
Earlier work this paper cites.
A new approach to causal inference in mortality studies with a sustained exposure period: application to control of the healthy worker survivor effect
J. Robins · 1986
Earlier work this paper cites.
On asymptotically efficient estimation in semiparametric models
A. Schick · 1986
Earlier work this paper cites.
Optimal replacement of gmc bus engines: An empirical model of harold zurcher
J. Rust · 1987
Earlier work this paper cites.
The analysis of randomized and non-randomized aids treatment trials using a new approach to causal inference in longitudinal studies
J. M. Robins · 1989
Earlier work this paper cites.
Optimal stopping and the american put
S. . Jacka · 1991
Earlier work this paper cites.
G-estimation of the effect of prophylaxis therapy for pneumocystis carinii pneumonia on the survival of aids patients
J. M. Robins, D. Blevins, G. Ritter, and M. Wulfsohn · 1992
Earlier work this paper cites.
Q-learning
C. J. Watkins and P. Dayan · 1992
Earlier work this paper cites.
The asymptotic variance of semiparametric estimators
W. K. Newey · 1994
Earlier work this paper cites.
Correcting for non-compliance in randomized trials using structural nested mean models
J. M. Robins · 1994
Earlier work this paper cites.
Estimation of regression coefficients when some regressors are not always observed
J. M. Robins, A. Rotnitzky, and L. P. Zhao · 1994
Earlier work this paper cites.
Sphere packing numbers for subsets of the boolean n-cube with bounded vapnik-chervonenkis dimension
D. Haussler · 1995
Earlier work this paper cites.
Semiparametric efficiency in multivariate regression models with missing data
J. M. Robins and A. Rotnitzky · 1995
Earlier work this paper cites.
Adjusting for nonignorable drop-out using semiparametric nonresponse models
D. O. Scharfstein, A. Rotnitzky, and J. M. Robins · 1999
Earlier work this paper cites.
A design for testing clinical strategies: biased adaptive within-subject randomization
P. W. Lavori and R. Dawson · 2000
Earlier work this paper cites.
Eligibility traces for off-policy policy evaluation
D. Precup · 2000
Earlier work this paper cites.
Marginal structural models and causal inference in epidemiology
J. M. Robins, M. A. Hernán, and B. Brumback · 2000
Earlier work this paper cites.
Marginal structural models to estimate the joint causal effect of nonrandomized treatments
M. A. Hernán, B. Brumback, and J. M. Robins · 2001
Earlier work this paper cites.
Optimal stopping and perpetual options for lévy processes
E. Mordecki · 2002
Earlier work this paper cites.
Kernel-based reinforcement learning
D. Ormoneit and Ś. Sen · 2002
Earlier work this paper cites.
On the sample complexity of reinforcement learning
S. M. Kakade · 2003
Earlier work this paper cites.
Error bounds for approximate policy iteration
R. Munos · 2003
Earlier work this paper cites.
Optimal dynamic treatment regimes
S. A. Murphy · 2003
Earlier work this paper cites.
Statistical treatment rules for heterogeneous populations
C. F. Manski · 2004
Earlier work this paper cites.
Optimal structural nested models for optimal sequential decisions
J. M. Robins · 2004
Earlier work this paper cites.
Tree-based batch mode reinforcement learning
D. Ernst, P. Geurts, and L. Wehenkel · 2005
Earlier work this paper cites.
A generalization error for q-learning
S. A. Murphy · 2005
Earlier work this paper cites.

Similar

Finite time bounds for sampling based fitted value iteration
C. Szepesvári and R. Munos · 2005
Cited alongside, same era.
Generalized marginal structural models for estimating optimal treatment regimes, 2006
L. Orellana, A. Rotnitzky, and J. Robins · 2006
Cited alongside, same era.
Large sample sieve estimation of semi-nonparametric models
X. Chen · 2007
Cited alongside, same era.
Demystifying optimal dynamic treatment regimes
E. E. Moodie, T. S. Richardson, and D. A. Stephens · 2007
Cited alongside, same era.
Causal effect models for realistic individualized treatment and intention to treat rules
M. J. van der Laan and M. L. Petersen · 2007
Cited alongside, same era.
Concentration of measure, 2008
J. Lafferty, H. Liu, and L. Wasserman · 2008
Cited alongside, same era.
Finite-time bounds for fitted value iteration
R. Munos and C. Szepesvári · 2008
Cited alongside, same era.
Estimation and extrapolation of optimal treatment and testing strategies
J. Robins, L. Orellana, and A. Rotnitzky · 2008
Cited alongside, same era.
Estimating response-maximized decision rules with applications to breastfeeding
E. E. Moodie, R. W. Platt, and M. S. Kramer · 2009
Cited alongside, same era.
Timing of initiation of antiretroviral therapy in aids-free hiv-1-infected patients: a collaborative analysis of 18 hiv cohort studies
When To Start Consortium · 2009
Cited alongside, same era.
Doubly robust policy evaluation and learning
M. Dudík, J. Langford, and L. Li · 2011
Cited alongside, same era.
Off-policy actor-critic
Original
T. Degris, M. White, and R. S. Sutton · 2012
Cited alongside, same era.

Then

Sample efficient policy search for optimal stopping domains
K. Goel, C. Dann, and E. Brunskill · 2017
Later among the works it cites.
Using options and covariance testing for long horizon off-policy policy evaluation
Z. Guo, P. S. Thomas, and E. Brunskill · 2017
Later among the works it cites.
Bootstrapping with models: Confidence intervals for off-policy evaluation
J. P. Hanna, P. Stone, and S. Niekum · 2017
Later among the works it cites.
Faster rates for policy learning
Original
A. Luedtke and A. Chambaz · 2017
Later among the works it cites.
Quasi-oracle estimation of heterogeneous treatment effects
Original
X. Nie and S. Wager · 2017
Later among the works it cites.
A reinforcement learning approach to weaning of mechanical ventilation in intensive care units
N. Prasad, L.-F. Cheng, C. Chivers, M. Draugelis, and B. E. Engelhardt · 2017
Later among the works it cites.
Reliable decision support using counterfactual models
P. Schulam and S. Saria · 2017
Later among the works it cites.
Residual weighted learning for estimating individualized treatment rules
X. Zhou, N. Mayer-Hamblett, U. Khan, and M. R. Kosorok · 2017
Later among the works it cites.
Design-based analysis in difference-in-differences settings with staggered adoption
Original
S. Athey and G. Imbens · 2018
Later among the works it cites.
More robust doubly robust off-policy evaluation
Original
M. Farajtabar, Y. Chow, and M. Ghavamzadeh · 2018
Later among the works it cites.
Balanced policy evaluation and learning
N. Kallus · 2018
Later among the works it cites.
Confounding-robust policy improvement
N. Kallus and A. Zhou · 2018
Later among the works it cites.
Who should be treated? empirical welfare maximization methods for treatment choice
T. Kitagawa and A. Tetenov · 2018
Later among the works it cites.
Q-learning with nearest neighbors
D. Shah and Q. Xie · 2018
Later among the works it cites.
Reinforcement learning: An introduction
R. S. Sutton and A. G. Barto · 2018
Later among the works it cites.
Targeted Learning in Data Science
M. J. Van der Laan and S. Rose · 2018
Later among the works it cites.
Estimation and inference of heterogeneous treatment effects using random forests
S. Wager and S. Athey · 2018
Later among the works it cites.
Interpretable dynamic treatment regimes
Y. Zhang, E. B. Laber, M. Davidian, and A. A. Tsiatis · 2018
Later among the works it cites.
Offline multi-action policy learning: Generalization and optimization
Original
Z. Zhou, S. Athey, and S. Wager · 2018
Later among the works it cites.
Generalized random forests
S. Athey, J. Tibshirani, and S. Wager · 2019
Closest in time.
Information-theoretic considerations in batch reinforcement learning
J. Chen and N. Jiang · 2019
Closest in time.
Double reinforcement learning for efficient off-policy evaluation in markov decision processes
Original
N. Kallus and M. Uehara · 2019
Closest in time.
Metalearners for estimating heterogeneous treatment effects using machine learning
S. R. Künzel, J. S. Sekhon, P. J. Bickel, and B. Yu · 2019
Closest in time.
Batch policy learning under constraints
H. Le, C. Voloshin, and Y. Yue · 2019
Closest in time.
Off-policy policy gradient with state distribution correction
Y. Liu, A. Swaminathan, A. Agarwal, and E. Brunskill · 2019
Closest in time.
Estimating dynamic treatment regimes in mobile health using v-learning
D. J. Luckett, E. B. Laber, A. R. Kahkoska, D. M. Maahs, E. Mayer-Davis, and M. R. Kosorok · 2019
Closest in time.
Dynamic Treatment Regimes: Statistical Methods for Precision Medicine
A. A. Tsiatis, M. Davidian, S. T. Holloway, and E. B. Laber · 2019
Closest in time.
Tighter problem-dependent regret bounds in reinforcement learning without domain knowledge using value function bounds
A. Zanette and E. Brunskill · 2019
Closest in time.
From predictive to prescriptive analytics
D. Bertsimas and N. Kallus · 2020
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…