2018

Learn to Interpret Atari Agents

Yang, Zhao, Bai, Song, Zhang, Li et al.

Understand

Deep reinforcement learning (DeepRL) agents surpass human-level performance in many tasks.

However, the direct mapping from states to actions makes it hard to interpret the rationale behind the decision-making of the agents.
In contrast to previous a-posteriori methods for visualizing DeepRL policies, in this work, we propose to equip the DeepRL model with an innate visualization ability.
Our proposed agent, named region-sensitive Rainbow (RS-Rainbow), is an end-to-end trainable network based on the original Rainbow, a powerful deep Q-network agent.

Built on

R. S. Sutton, “Learning to predict by the methods of temporal differences,” Machine Learning , vol. 3, no. 1, pp. 9–44, 1988
1988
Earlier work this paper cites.
G. Tesauro, “Temporal difference learning and td-gammon,” Communications of the ACM , vol. 38, no. 3, pp. 58–68, 1995
1995
Earlier work this paper cites.
R. S. Sutton and A. G. Barto, Introduction to Reinforcement Learning , 1st ed. MIT Press, 1998
1998
Earlier work this paper cites.
F. Elizalde, L. Sucar, M. Luque, F. Díez, and A. Reyes Ballesteros, “Policy explanation in factored markov decision processes,” in European Workshop on Probabilistic Graphical Models , 2008
2008
Earlier work this paper cites.
L. v. d. Maaten and G. Hinton, “Visualizing data using t-sne,” Journal of Machine Learning Research , vol. 9, pp. 2579–2605, 2008
2008
Earlier work this paper cites.
O. Z. Khan, P. Poupart, and J. P. Black, “Minimal sufficient explanations for factored markov decision processes,” in International Conference on Automated Planning and Scheduling , 2009
2009
Earlier work this paper cites.
V. Nair and G. E. Hinton, “Rectified linear units improve restricted boltzmann machines,” in Proceedings of the 27th International Conference on Machine Learning (ICML) , 2010
2010
Earlier work this paper cites.
T. Dodson, N. Mattei, and J. Goldsmith, “A natural language argumentation interface for explanation generation in markov decision processes,” in Algorithmic Decision Theory , 2011
2011
Earlier work this paper cites.
A. Krizhevsky, I. Sutskever, and G. E. Hinton, “Imagenet classification with deep convolutional neural networks,” in Advances in Neural Information Processing Systems (NeurIPS) , 2012
2012
Earlier work this paper cites.
E. Todorov, T. Erez, and Y. Tassa, “Mujoco: A physics engine for model-based control,” IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) , pp. 5026–5033, 2012
2012
Earlier work this paper cites.
V. Mnih, K. Kavukcuoglu, D. Silver, A. Graves, I. Antonoglou, D. Wierstra, and M. Riedmiller, “Playing atari with deep reinforcement learning,” in Advances in Neural Information Processing Systems (NeurIPS) Deep Learning Workshop , 2013
2013
Earlier work this paper cites.
M. G. Bellemare, Y. Naddaf, J. Veness, and M. Bowling, “The arcade learning environment: An evaluation platform for general agents,” Journal of Artificial Intelligence Research , vol. 47, pp. 253–279, 2013
2013
Earlier work this paper cites.
K. Simonyan, A. Vedaldi, and A. Zisserman, “Deep inside convolutional networks: Visualising image classification models and saliency maps,” in International Conference on Learning Representations (ICLR) , 2014
2014
Earlier work this paper cites.
M. D. Zeiler and R. Fergus, “Visualizing and understanding convolutional networks,” in The European Conference on Computer Vision (ECCV) , 2014
2014
Earlier work this paper cites.
K. Simonyan and A. Zisserman, “Very deep convolutional networks for large-scale image recognition,” arXiv preprint arXiv:1409.1556 , 2014
Original
2014
Earlier work this paper cites.
J. T. Springenberg, A. Dosovitskiy, T. Brox, and M. Riedmiller, “Striving for simplicity: The all convolutional net,” in International Conference on Learning Representations (ICLR) , 2015
2015
Earlier work this paper cites.
V. Mnih, K. Kavukcuoglu, D. Silver, A. A. Rusu, J. Veness, M. G. Bellemare, A. Graves, M. Riedmiller, A. K. Fidjeland, G. Ostrovski, S. Petersen, C. Beattie, A. Sadik, I. Antonoglou, H. King, D. Kumaran, D. Wierstra, S. Legg, and D. Hassabis, “Human-level control through deep reinforcement learning,” Nature , vol. 518, no. 7540, pp. 529–533, 2015
2015
Earlier work this paper cites.
I. Sorokin, A. Seleznev, M. Pavlov, A. Fedorov, and A. Ignateva, “Deep attention recurrent q-network,” in Deep Reinforcement Learning Workshop, Advances in Neural Information Processing Systems (NeurIPS) , 2015
2015
Earlier work this paper cites.
K. Xu, J. Ba, R. Kiros, K. Cho, A. Courville, R. Salakhutdinov, R. Zemel, and Y. Bengio, “Show, attend and tell: Neural image caption generation with visual attention,” in Proceedings of the 32nd International Conference on Machine Learning (ICML) , 2015
2015
Earlier work this paper cites.

Similar

M. Hausknecht and P. Stone, “Deep recurrent q-learning for partially observable mdps,” in 2015 AAAI Fall Symposium Series , 2015
2015
Cited alongside, same era.
D. Bahdanau, K. Cho, and Y. Bengio, “Neural machine translation by jointly learning to align and translate,” in International Conference on Learning Representations (ICLR) , 2015
2015
Cited alongside, same era.
T. P. Lillicrap, J. J. Hunt, A. Pritzel, N. Heess, T. Erez, Y. Tassa, D. Silver, and D. Wierstra, “Continuous control with deep reinforcement learning,” arXiv preprint arXiv:1509.02971 , 2015
Original
2015
Cited alongside, same era.
J. Zhang, M. Wang, S. Zhang, X. Li, and X. Wu, “Spatiochromatic context modeling for color saliency analysis,” IEEE Transactions on Neural Networks and Learning Systems (TNNLS) , vol. 27, no. 6, pp. 1177–1189, 2016
2016
Cited alongside, same era.
M. Kempka, M. Wydmuch, G. Runc, J. Toczek, and W. Jaśkowski, “Vizdoom: A doom-based ai research platform for visual reinforcement learning,” in IEEE Conference on Computational Intelligence and Games (CIG) , 2016
2016
Cited alongside, same era.
Z. Wang, T. Schaul, M. Hessel, H. Hasselt, M. Lanctot, and N. Freitas, “Dueling network architectures for deep reinforcement learning,” in Proceedings of the 33rd International Conference on Machine Learning (ICML) , 2016
2016
Cited alongside, same era.
T. Zahavy, N. Ben-Zrihem, and S. Mannor, “Graying the black box: Understanding dqns,” in Proceedings of the 33rd International Conference on Machine Learning (ICML) , 2016
2016
Cited alongside, same era.
H. van Hasselt, A. Guez, and D. Silver, “Deep reinforcement learning with double q-learning,” in Proceedings of the 30th AAAI Conference on Artificial Intelligence (AAAI) , 2016
2016
Cited alongside, same era.
T. Schaul, J. Quan, I. Antonoglou, and D. Silver, “Prioritized experience replay,” in International Conference on Learning Representations (ICLR) , 2016
2016
Cited alongside, same era.
V. Mnih, A. P. Badia, M. Mirza, A. Graves, T. Lillicrap, T. Harley, D. Silver, and K. Kavukcuoglu, “Asynchronous methods for deep reinforcement learning,” in Proceedings of the 33rd International Conference on Machine Learning (ICML) , 2016
2016
Cited alongside, same era.
D.-A. Clevert, T. Unterthiner, and S. Hochreiter, “Fast and accurate deep network learning by exponential linear units (elus),” in International Conference on Learning Representations (ICLR) , 2016
2016
Cited alongside, same era.
M. Wang, W. Fu, S. Hao, D. Tao, and X. Wu, “Scalable semi-supervised learning by efficient anchor graph regularization,” IEEE Transactions on Knowledge and Data Engineering (TKDE) , vol. 28, no. 7, pp. 1864–1877, 2016
2016
Cited alongside, same era.

Then

A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin, “Attention is all you need,” in Advances in Neural Information Processing Systems (NeurIPS) , 2017
2017
Later among the works it cites.
J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov, “Proximal policy optimization algorithms,” CoRR , vol. abs/1707.06347, 2017
Original
2017
Later among the works it cites.
W. Wang, J. Shen, R. Yang, and F. Porikli, “Saliency-aware video object segmentation,” IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI) , vol. 40, no. 1, pp. 20–33, 2018
2018
Closest in time.
W. Wang, J. Shen, and L. Shao, “Video salient object detection via fully convolutional networks,” TIP , vol. 27, no. 1, pp. 38–49, 2018
2018
Closest in time.
S. Greydanus, A. Koul, J. Dodge, and A. Fern, “Visualizing and understanding Atari agents,” in Proceedings of the 35th International Conference on Machine Learning (ICML) , 2018
2018
Closest in time.
M. Hessel, J. Modayil, H. Van Hasselt, T. Schaul, G. Ostrovski, W. Dabney, D. Horgan, B. Piot, M. Azar, and D. Silver, “Rainbow: Combining improvements in deep reinforcement learning,” in Proceedings of the 32nd AAAI Conference on Artificial Intelligence (AAAI) , 2018
2018
Closest in time.
M. Fortunato, M. G. Azar, B. Piot, J. Menick, M. Hessel, I. Osband, A. Graves, V. Mnih, R. Munos, D. Hassabis, O. Pietquin, C. Blundell, and S. Legg, “Noisy networks for exploration,” in International Conference on Learning Representations (ICLR) , 2018
2018
Closest in time.
Y. Yuan and J. Wang, “Ocnet: Object context network for scene parsing,” arXiv preprint arXiv:1809.00916 , 2018
Original
2018
Closest in time.
X. Wang, R. Girshick, A. Gupta, and K. He, “Non-local neural networks,” in The IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , 2018
2018
Closest in time.
A. Nichol, V. Pfau, C. Hesse, O. Klimov, and J. Schulman, “Gotta learn fast: A new benchmark for generalization in rl,” arXiv preprint arXiv:1804.03720 , 2018
Original
2018
Closest in time.
L.-C. Chen, G. Papandreou, I. Kokkinos, K. Murphy, and A. L. Yuille, “Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs,” IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI) , vol. 40, no. 4, pp. 834–848, 2018
2018
Closest in time.
D. Horgan, J. Quan, D. Budden, G. Barth-Maron, M. Hessel, H. van Hasselt, and D. Silver, “Distributed prioritized experience replay,” in International Conference on Learning Representations (ICLR) , 2018
2018
Closest in time.
G. Zhu, L. Zhang, L. Yang, L. Mei, S. Shah, M. Bennamoun, and P. Shen, “Redundancy and attention in convolutional lstm for gesture recognition,” IEEE Transactions on Neural Networks and Learning Systems (TNNLS) , pp. 1–13, 2019
2019
Closest in time.
L. Yang, Q. Song, Y. Wu, and M. Hu, “Attention inspiring receptive-fields network for learning invariant representations,” IEEE Transactions on Neural Networks and Learning Systems (TNNLS) , vol. 30, no. 6, pp. 1744–1755, 2019
2019
Closest in time.
C. Berner, G. Brockman, B. Chan, V. Cheung, P. Debiak, C. Dennison, D. Farhi, Q. Fischer, S. Hashme, C. Hesse et al. , “Dota 2 with large scale deep reinforcement learning,” arXiv preprint arXiv:1912.06680 , 2019
Original
2019
Closest in time.
O. Vinyals, I. Babuschkin, W. M. Czarnecki, M. Mathieu, A. Dudzik, J. Chung, D. H. Choi, R. Powell, T. Ewalds, P. Georgiev et al. , “Grandmaster level in starcraft ii using multi-agent reinforcement learning,” Nature , vol. 575, no. 7782, pp. 350–354, 2019
2019
Closest in time.
M. Jaderberg, W. M. Czarnecki, I. Dunning, L. Marris, G. Lever, A. G. Castañeda, C. Beattie, N. C. Rabinowitz, A. S. Morcos, A. Ruderman, N. Sonnerat, T. Green, L. Deason, J. Z. Leibo, D. Silver, D. Hassabis, K. Kavukcuoglu, and T. Graepel, “Human-level performance in 3d multiplayer games with population-based reinforcement learning,” Science , vol. 364, no. 6443, pp. 859–865, 2019
2019
Closest in time.
N. Brown and T. Sandholm, “Superhuman ai for multiplayer poker,” Science , vol. 365, no. 6456, pp. 885–890, 2019
2019
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…