2019

Explain Your Move: Understanding Agent Actions Using Specific and Relevant Feature Attribution

Puri, Nikaash, Verma, Sukriti, Gupta, Piyush et al.

Understand

As deep reinforcement learning (RL) is applied to more tasks, there is a need to visualize and understand the behavior of learned agents.

Saliency maps explain agent behavior by highlighting the features of the input state that are most relevant for the agent in taking an action.
Existing perturbation-based approaches to compute saliency often highlight regions of the input that are not relevant to the action taken by the agent.
Our proposed approach, SARFA (Specific and Relevant Feature Attribution), generates more focused saliency maps by balancing two aspects (specificity and relevance) that capture different desiderata of saliency.

Built on

Policy explanation in factored markov decision processes
Francisco Elizalde, L Enrique Sucar, Manuel Luque, J Diez, and Alberto Reyes · 2008
Earlier work this paper cites.
Visualizing data using t-sne
Laurens van der Maaten and Geoffrey Hinton · 2008
Earlier work this paper cites.
Minimal sufficient explanations for factored markov decision processes
Omar Zia Khan, Pascal Poupart, and James P Black · 2009
Earlier work this paper cites.
Recurrent neural network based language model
Tomáš Mikolov, Martin Karafiát, Lukáš Burget, Jan Černockỳ, and Sanjeev Khudanpur · 2010
Earlier work this paper cites.
A natural language argumentation interface for explanation generation in markov decision processes
Thomas Dodson, Nicholas Mattei, and Judy Goldsmith · 2011
Earlier work this paper cites.
Imagenet classification with deep convolutional neural networks
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton · 2012
Earlier work this paper cites.
Deep inside convolutional networks: Visualising image classification models and saliency maps
Original
Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman · 2013
Earlier work this paper cites.
Striving for simplicity: The all convolutional net
Original
Jost Tobias Springenberg, Alexey Dosovitskiy, Thomas Brox, and Martin Riedmiller · 2014
Earlier work this paper cites.
Visualizing and understanding convolutional networks
Matthew D Zeiler and Rob Fergus · 2014
Earlier work this paper cites.
Visualizing and understanding recurrent networks
Original
Andrej Karpathy, Justin Johnson, and Li Fei-Fei · 2015
Earlier work this paper cites.

Similar

Human-level control through deep reinforcement learning
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al · 2015
Cited alongside, same era.
Deep residual learning for image recognition
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun · 2016
Cited alongside, same era.
Asynchronous methods for deep reinforcement learning
Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu · 2016
Cited alongside, same era.
Why should i trust you?: Explaining the predictions of any classifier
Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin · 2016
Cited alongside, same era.
Mastering the game of go with deep neural networks and tree search
David Silver, Aja Huang, Chris J Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schrittwieser, Ioannis Antonoglou, Veda Panneershelvam, Marc Lanctot, et al · 2016
Cited alongside, same era.
Graying the black box: Understanding dqns
Tom Zahavy, Nir Ben-Zrihem, and Shie Mannor · 2016
Cited alongside, same era.
Interpretable explanations of black boxes by meaningful perturbation
Ruth C Fong and Andrea Vedaldi · 2017
Cited alongside, same era.
Improving robot controller transparency through autonomous policy explanation
Bradley Hayes and Julie A Shah · 2017
Cited alongside, same era.
A unified approach to interpreting model predictions
Scott M Lundberg and Su-In Lee · 2017
Cited alongside, same era.
Grad-cam: Visual explanations from deep networks via gradient-based localization
Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra · 2017
Cited alongside, same era.

Then

Learning important features through propagating activation differences
Avanti Shrikumar, Peyton Greenside, and Anshul Kundaje · 2017
Later among the works it cites.
Mastering the game of go without human knowledge
David Silver, Julian Schrittwieser, Karen Simonyan, Ioannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, et al · 2017
Later among the works it cites.
Axiomatic attribution for deep networks
Mukund Sundararajan, Ankur Taly, and Qiqi Yan · 2017
Later among the works it cites.
Grad-cam++: Generalized gradient-based visual explanations for deep convolutional networks
Aditya Chattopadhay, Anirban Sarkar, Prantik Howlader, and Vineeth N Balasubramanian · 2018
Later among the works it cites.
Visualizing and understanding atari agents
Samuel Greydanus, Anurag Koul, Jonathan Dodge, and Alan Fern · 2018
Later among the works it cites.
Transparency and explanation in deep reinforcement learning neural networks
Original
Rahul Iyer, Yuezhang Li, Huao Li, Michael Lewis, Ramitha Sundar, and Katia P. Sycara · 2018
Later among the works it cites.
Anchors: High-precision model-agnostic explanations
Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin · 2018
Later among the works it cites.
Top-down neural attention by excitation backprop
Jianming Zhang, Sarah Adel Bargal, Zhe Lin, Jonathan Brandt, Xiaohui Shen, and Stan Sclaroff · 2018
Later among the works it cites.
Interpretation of neural networks is fragile
Amirata Ghorbani, Abubakar Abid, and James Zou · 2019
Closest in time.
Global aggregations of local explanations for black box models
Ilse van der Linden, Hinda Haned, and Evangelos Kanoulas · 2019
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…