2020

Distal Explanations for Model-free Explainable Reinforcement Learning

Madumal, Prashan, Miller, Tim, Sonenberg, Liz et al.

Understand

In this paper we introduce and evaluate a distal explanation model for model-free reinforcement learning agents that can generate explanations for `why' and `why not' questions.

Our starting point is the observation that causal models can generate opportunity chains that take the form of `A enables B and B causes C'.
Using insights from an analysis of 240 explanations generated in a human-agent experiment, we define a distal explanation model that can analyse counterfactuals and opportunity chains using decision trees and causal models.
A recurrent neural network is employed to learn opportunity chains, and decision trees are used to improve the accuracy of task prediction and the generated counterfactuals.

Built on

Conversational processes and causal explanation
Denis J Hilton · 1990
Earlier work this paper cites.
Input generalization in delayed reinforcement learning: An algorithm and performance comparisons
David Chapman and Leslie Pack Kaelbling · 1991
Earlier work this paper cites.
Self-improving reactive agents based on reinforcement learning, planning and teaching
Long-Ji Lin · 1992
Earlier work this paper cites.
Agency and causal explanation
Jennifer Hornsby · 1993
Earlier work this paper cites.
Exploiting structure in policy construction
Craig Boutilier, Richard Dearden, Moises Goldszmidt, et al · 1995
Earlier work this paper cites.
For you can’t always get what you want: When preconditions are better explanations than goals
John McClure and Denis Hilton · 1997
Earlier work this paper cites.
Bidirectional recurrent neural networks
Mike Schuster and Kuldip K Paliwal · 1997
Earlier work this paper cites.
Causes and explanations: A structural-model approach. part II: Explanations
Joseph Y Halpern and Judea Pearl · 2005
Earlier work this paper cites.
The course of events: counterfactuals, causal sequences, and explanation
Denis J Hilton, John L McClure, and Ben R. Slugoski · 2005
Earlier work this paper cites.
Causal models: How people think about the world and its alternatives
Steven Sloman · 2005
Earlier work this paper cites.
Using thematic analysis in psychology
Virginia Braun and Victoria Clarke · 2006
Earlier work this paper cites.
An MDP approach for explanation generation
Francisco Elizalde, Luis Enrique Sucar, Alberto Reyes, and Pablo Debuen · 2007
Earlier work this paper cites.
Judgments of voluntary and physical causes in causal chains: Probabilistic and social functionalist criteria for attributions
John McClure, Denis J Hilton, and Robbie M Sutton · 2007
Earlier work this paper cites.
Policy explanation in factored markov decision processes
Francisco Elizalde, L Enrique Sucar, Manuel Luque, J Diez, and Alberto Reyes · 2008
Earlier work this paper cites.
Expert evaluation of probabilistic explanations
Francisco Elizalde and Luis Enrique Sucar · 2009
Earlier work this paper cites.

Similar

Minimal sufficient explanations for factored markov decision processes
Omar Zia Khan, Pascal Poupart, and James P Black · 2009
Cited alongside, same era.
Why and why not explanations improve the intelligibility of context-aware intelligent systems
Brian Y Lim, Anind K Dey, and Daniel Avrahami · 2009
Cited alongside, same era.
Selecting explanations from causal chains: Do statistical principles explain preferences for voluntary causes?
Denis J Hilton, John McClure, and Robbie M Sutton · 2010
Cited alongside, same era.
Amazon’s mechanical turk: A new source of inexpensive, yet high-quality, data?
Michael Buhrmester, Tracy Kwang, and Samuel D Gosling · 2011
Cited alongside, same era.
Between-or within-subjects design: A methodological dilemma
Gideon Keren · 2014
Cited alongside, same era.
How people explain their own and others’ behavior: a theory of lay causal explanations
Gisela Böhm and Hans-Rüdiger Pfister · 2015
Cited alongside, same era.
OpenAI Gym, 2016
Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba · 2016
Cited alongside, same era.
Explanations in causal chains: Selecting distal causes requires exportable mechanisms
Jonas Nagel and Simon Stephan · 2016
Cited alongside, same era.
Trust calibration within a human-robot team: Comparing automatically generated explanations
Ning Wang, David V Pynadath, and Susan G Hill · 2016
Cited alongside, same era.
How people explain action (and autonomous intelligent systems should too)
Maartje MA De Graaf and Bertram F Malle · 2017
Cited alongside, same era.
Autonomous self-explanation of behavior for interactive reinforcement learning agents
Yosuke Fukuchi, Masahiko Osawa, Hiroshi Yamakawa, and Michita Imai · 2017
Cited alongside, same era.
Improving robot controller transparency through autonomous policy explanation
Bradley Hayes and Julie A Shah · 2017
Cited alongside, same era.

Then

Trends and trajectories for explainable, accountable and intelligible systems: An HCI research agenda
Ashraf Abdul, Jo Vermeulen, Danding Wang, Brian Y Lim, and Mohan Kankanhalli · 2018
Later among the works it cites.
Highlights: Summarizing agent behavior to people
Dan Amir and Ofra Amir · 2018
Later among the works it cites.
Metrics for explainable AI: Challenges and prospects
Original
Robert R Hoffman, Shane T Mueller, Gary Klein, and Jordan Litman · 2018
Later among the works it cites.
Explaining explanation, part 3: The causal landscape
Gary Klein · 2018
Later among the works it cites.
Contrastive explanations for reinforcement learning in terms of expected consequences
Original
Jasper van der Waa, Jurriaan van Diggelen, Karel van den Bosch, and Mark Neerincx · 2018
Later among the works it cites.
Counterfactuals in explainable artificial intelligence (XAI): evidence from human reasoning
Ruth MJ Byrne · 2019
Later among the works it cites.
Darpa’s explainable artificial intelligence program
David Gunning and David W Aha · 2019
Later among the works it cites.
Explainable reinforcement learning via reward decomposition
Zoe Juozapaitis, Anurag Koul, Alan Fern, Martin Erwig, and Finale Doshi-Velez · 2019
Later among the works it cites.
Conservative q-improvement: Reinforcement learning for an interpretable decision-tree policy
Original
Aaron M Roth, Nicholay Topin, Pooyan Jamshidi, and Manuela Veloso · 2019
Later among the works it cites.
Cxplain: Causal explanations for model interpretation under uncertainty
Patrick Schwab and Walter Karlen · 2019
Later among the works it cites.
Autonomous generation of robust and focused explanations for robot policies
Oliver Struckmeier, Mattia Racca, and Ville Kyrki · 2019
Later among the works it cites.
Explanation-based reward coaching to improve human performance via reinforcement learning
Aaquib Tabrez, Shivendra Agrawal, and Bradley Hayes · 2019
Later among the works it cites.
Generation of policy-level explanations for reinforcement learning
Nicholay Topin and Manuela Veloso · 2019
Later among the works it cites.
Designing theory-driven user-centric explainable ai
Danding Wang, Qian Yang, Ashraf Abdul, and Brian Y Lim · 2019
Later among the works it cites.
Explainable reinforcement learning through a causal lens
Prashan Madumal, Tim Miller, Liz Sonenberg, and Frank Vetere · 2020
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…