2017

Visualizing and Understanding Atari Agents

Greydanus, Sam, Koul, Anurag, Dodge, Jonathan et al.

Understand

While deep reinforcement learning (deep RL) agents are effective at maximizing rewards, it is often unclear what strategies they use to do so.

In this paper, we take a step toward explaining deep RL agents through a case study using Atari 2600 environments.
In particular, we focus on using saliency maps to understand how an agent learns and executes a policy.
We introduce a method for generating useful saliency maps and use it to show 1) what strong agents attend to, 2) whether agents are making decisions for the right or wrong reasons, and 3) how agents evolve during learning.

Built on

Policy Explanation in Factored Markov Decision Processes
Elizalde, F., Sucar, L. E., Luque, M., Díez, F. J., and Reyes, A · 2008
Earlier work this paper cites.
Minimal Sufficient Explanations for Factored Markov Decision Processes
Khan, O. Z., Poupart, P., and Black, J. P · 2009
Earlier work this paper cites.
Recurrent neural network based language model
Mikolov, T., Karafiát, M., Burget, L., and Khudanpur, S · 2010
Earlier work this paper cites.
A Natural Language Argumentation Interface for Explanation Generation in Markov Decision Processes
Dodson, T., Mattei, N., and Goldsmith, J · 2011
Earlier work this paper cites.
ImageNet Classification with Deep Convolutional Neural Networks
Krizhevsky, A., Sutskever, I., and Hinton, G. E · 2012
Earlier work this paper cites.
The Arcade Learning Environment: An Evaluation Platform for General Agents
Bellemare, M. G., Naddaf, Y., Veness, J., and Bowling, M · 2013
Earlier work this paper cites.
Multi-task Neural Networks for QSAR Predictions
Original
Dahl, G. E., Jaitly, N., and Salakhutdinov, R · 2014
Earlier work this paper cites.
Adam: A Method for Stochastic Optimization
Original
Kingma, D. P. and Ba, J. L · 2014
Earlier work this paper cites.
Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps
Original
Simonyan, K., Vedaldi, A., and Zisserman, A · 2014
Earlier work this paper cites.
Visualizing and Understanding Convolutional Networks
Original
Zeiler, M. D. and Fergus, R · 2014
Earlier work this paper cites.

Similar

Deep Residual Learning for Image Recognition
Original
He, K., Zhang, X., Ren, S., and Sun, J · 2015
Cited alongside, same era.
Deep Visual-Semantic Alignments for Generating Image Descriptions
Original
Karpathy, A. and Fei-Fei, L · 2015
Cited alongside, same era.
Human-level control through deep reinforcement learning
Mnih, V., Kavukcuoglu, K., Silver, D., Rusu, A. A., Veness, J., Bellemare, M. G., Graves, A., Riedmiller, M., Fidjeland, A. K., Ostrovski, G., Petersen, S., Beattie, C., Sadik, A., Antonoglou, I., King, H., Kumaran, D., Wierstra, D., Legg, S., and Hassabis, D · 2015
Cited alongside, same era.
Striving for Simplicity: The All Convolutional Net
Original
Springenberg, J. T., Dosovitskiy, A., Brox, T., and Riedmiller, M · 2015
Cited alongside, same era.
OpenAI Gym
Original
Brockman, G., Cheung, V., Pettersson, L., Schneider, J., Schulman, J., Tang, J., and Zaremba, W · 2016
Cited alongside, same era.
Visualizing and Understanding Recurrent Networks
Original
Karpathy, A., Johnson, J., and Fei-Fei, L · 2016
Cited alongside, same era.
Asynchronous Methods for Deep Reinforcement Learning
Original
Mnih, V., Puigdomènech, A., Mirza, M., Graves, A., Harley, T., Lillicrap, T. P., Silver, D., and Kavukcuoglu, K · 2016
Cited alongside, same era.
”Why Should I Trust You?”: Explaining the Predictions of Any Classifier
Original
Ribeiro, M. T., Singh, S., and Guestrin, C · 2016
Cited alongside, same era.
High Dimensional Continuous Control Using Generalized Advantage Estimation
Original
Schulman, J., Moritz, P., Levine, S., Jordan, M. I., and Abbeel, P · 2016
Cited alongside, same era.

Then

Dueling Network Architectures for Deep Reinforcement Learning
Original
Wang, Z., Schaul, T., Hessel, M., Van Hasselt, H., and Lanctot, M · 2016
Later among the works it cites.
Graying the black box: Understanding DQNs
Original
Zahavy, T., Baram, N., and Mannor, S · 2016
Later among the works it cites.
Top-down Neural Attention by Excitation Backprop
Original
Zhang, J., Lin, Z., Brandt, J., Shen, X., and Sclaroff, S · 2016
Later among the works it cites.
Real Time Image Saliency for Black Box Classifiers
Original
Dabkowski, P. and Gal, Y · 2017
Closest in time.
Interpretable Explanations of Black Boxes by Meaningful Perturbation
Original
Fong, R. C. and Vedaldi, A · 2017
Closest in time.
Improving Robot Controller Transparency Through Autonomous Policy Explanation
Hayes, B. and Shah, J. A · 2017
Closest in time.
Automatic Rule Extraction from Long Short Term Memory Networks
Original
Murdoch, W. J. and Szlam, A · 2017
Closest in time.
Learning Important Features Through Propagating Activation Differences
Shrikumar, A., Greenside, P., and Kundaje, A · 2017
Closest in time.
Mastering the game of Go without human knowledge
Silver, D., Schrittwieser, J., Simonyan, K., Antonoglou, I., Huang, A., Guez, A., Hubert, T., Baker, L., Lai, M., Bolton, A., Chen, Y., Lillicrap, T., Hui, F., Sifre, L., van den Driessche, G., Graepel, T., and Hassabis, D · 2017
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…