2016

Bootstrapping with Models: Confidence Intervals for Off-Policy Evaluation

Hanna, Josiah P., Stone, Peter, Niekum, Scott

Understand

For an autonomous agent, executing a poor policy may be costly or even dangerous.

For such agents, it is desirable to determine confidence interval lower bounds on the performance of any given policy without executing said policy.
Current methods for exact high confidence off-policy evaluation that use importance sampling require a substantial amount of data to achieve a tight lower bound.
Existing model-based methods only address the problem in discrete state spaces.

Bootstrap methods: Another look at the jackknife
B et al. Efron · 1979
Earlier work this paper cites.
Better bootstrap confidence intervals
Bradley Efron · 1987
Earlier work this paper cites.
Reinforcement Learning: An Introduction
Richard S. Sutton and Andrew G. Barto · 1998
Earlier work this paper cites.
Bootstrap confidence intervals: when, which, what? a practical guide for medical statisticians
James Carpenter and John Bithell · 2000
Earlier work this paper cites.
Eligibility traces for off-policy policy evaluation
Doina Precup, Richard S. Sutton, and Satinder Singh · 2000
Earlier work this paper cites.
Near-optimal reinforcement learning in polynomial time
Michael Kearns and Satinder Singh · 2002
Earlier work this paper cites.
Coronary heart disease risk prediction in the atherosclerosis risk in communities (aric) study
Lloyd E Chambless, Aaron R Folsom, A Richey Sharrett, Paul Sorlie, David Couper, Moyses Szklo, and F Javier Nieto · 2003
Earlier work this paper cites.

Reinforcement learning in finite mdps: Pac analysis
Alexander L Strehl, Lihong Li, and Michael L Littman · 2009
Cited alongside, same era.
Real time targeted exploration in large domains
Todd Hester and Peter Stone · 2010
Cited alongside, same era.
Interval estimation for reinforcement-learning algorithms in continuous-state domains
Original
Martha White and Adam White · 2010
Cited alongside, same era.
Doubly robust policy evaluation and learning
Miroslav Dudík, John Langford, and Lihong Li · 2011
Cited alongside, same era.
Agnostic system identification for model-based reinforcement learning
Stephane Ross and J. Andrew Bagnell · 2012
Cited alongside, same era.
Counterfactual reasoning and learning systems: the example of computational advertising
Léon Bottou, Jonas Peters, Joaquin Quinonero Candela, Denis Xavier Charles, Max Chickering, Elon Portugaly, Dipankar Ray, Patrice Y Simard, and Ed Snelson · 2013
Cited alongside, same era.
Off-policy Evaluation in Markov Decision Processes
Cosmin Paduraru · 2013
Cited alongside, same era.

Probably approximately correct mdp learning and control with temporal logic constraints
Jie Fu and Ufuk Topcu · 2014
Later among the works it cites.
Robust policy optimization with baseline guarantees
Original
Yinlam Chow, Marek Petrik, and Mohammad Ghavamzadeh · 2015
Later among the works it cites.
Doubly robust off-policy evaluation for reinforcement learning
Nan Jiang and Lihong Li · 2015
Later among the works it cites.
High confidence off-policy evaluation
P. S. Thomas, Georgios Theocharous, and Mohammad Ghavamzadeh · 2015
Later among the works it cites.
High confidence policy improvement
P. S. Thomas, Georgios Theocharous, and Mohammad Ghavamzadeh · 2015
Later among the works it cites.
Safe Reinforcement Learning
P.S. Thomas · 2015
Later among the works it cites.
Data-efficient off-policy policy evaluation for reinforcement learning
P.S. Thomas and Emma Brunskill · 2016
Closest in time.

alphaXiv searches the wider corpus for related work and actual follow-ups.

alphaXiv is searching for related work…

Reading the bibliography…