2020

Demonstration-efficient Inverse Reinforcement Learning in Procedurally Generated Environments

Sestini, Alessandro, Kuhnle, Alexander, Bagdanov, Andrew D.

Understand

Deep Reinforcement Learning achieves very good results in domains where reward functions can be manually engineered.

At the same time, there is growing interest within the community in using games based on Procedurally Content Generation (PCG) as benchmark environments since this type of environment is perfect for studying overfitting and generalization of agents under domain shift.
Inverse Reinforcement Learning (IRL) can instead extrapolate reward functions from expert demonstrations, with good results even on high-dimensional problems, however there are no examples of applying these techniques to procedurally-generated environments.
This is mostly due to the number of demonstrations needed to find a good reward model.

Built on

Expert-Level Atari Imitation Learning from Demonstrations Only
Original
Cai, X.-Q.; Ding, Y.-X.; Jiang, Y.; and Zhou, Z.-H. 2019 · 1909
Earlier work this paper cites.
Procedural content generation: from automatically generating game levels to increasing generality in machine learning
Original
Risi, S.; and Togelius, J. 2019 · 1911
Earlier work this paper cites.
Leveraging procedural generation to benchmark reinforcement learning
Original
Cobbe, K.; Hesse, C.; Hilton, J.; and Schulman, J. 2019 · 1912
Earlier work this paper cites.
Dota 2 with Large Scale Deep Reinforcement Learning
OpenAI; Berner, C.; Brockman, G.; Chan, B.; Cheung, V.; Debiak, P.; Dennison, C.; Farhi, D.; Fischer, Q.; Hashme, S.; Hesse, C.; Józefowicz, R.; Gray, S.; Olsson, C.; Pachocki, J.; Petrov, M.; de Oliveira Pinto, H. P.; Raiman, J.; Salimans, T.; Schlatter, J.; Schneider, J.; Sidor, S.; Sutskever, I.; Tang, J.; Wolski, F.; and Zhang, S. 2019 · 1912
Earlier work this paper cites.
A Framework for Behavioural Cloning
Bain, M.; and Sammut, C. 1995 · 1995
Earlier work this paper cites.
Algorithms for Inverse Reinforcement Learning
Ng, A. Y.; and Russell, S. 2000 · 2000
Earlier work this paper cites.
The NetHack Learning Environment
Original
Küttler, H.; Nardelli, N.; Miller, A. H.; Raileanu, R.; Selvatici, M.; Grefenstette, E.; and Rocktäschel, T. 2020 · 2006
Earlier work this paper cites.
A game-theoretic approach to apprenticeship learning
Syed, U.; and Schapire, R. E. 2008 · 2008
Earlier work this paper cites.
Maximum entropy inverse reinforcement learning
Ziebart, B. D.; Maas, A.; Bagnell, J. A.; and Dey, A. K. 2008 · 2008
Earlier work this paper cites.
Interactively shaping agents via human reinforcement: The TAMER framework
Knox, W. B.; and Stone, P. 2009 · 2009
Earlier work this paper cites.
A reduction of imitation learning and structured prediction to no-regret online learning
Ross, S.; Gordon, G.; and Bagnell, D. 2011 · 2011
Earlier work this paper cites.
Mujoco: A physics engine for model-based control
Todorov, E.; Erez, T.; and Tassa, Y. 2012 · 2012
Earlier work this paper cites.

Similar

The arcade learning environment: An evaluation platform for general agents
Bellemare, M. G.; Naddaf, Y.; Veness, J.; and Bowling, M. 2013 · 2013
Cited alongside, same era.
Generative adversarial nets
Goodfellow, I.; Pouget-Abadie, J.; Mirza, M.; Xu, B.; Warde-Farley, D.; Ozair, S.; Courville, A.; and Bengio, Y. 2014 · 2014
Cited alongside, same era.
Human-level control through deep reinforcement learning
Mnih, V.; Kavukcuoglu, K.; Silver, D.; Rusu, A. A.; Veness, J.; Bellemare, M. G.; Graves, A.; Riedmiller, M.; Fidjeland, A. K.; Ostrovski, G.; et al. 2015 · 2015
Cited alongside, same era.
Trust region policy optimization
Schulman, J.; Levine, S.; Abbeel, P.; Jordan, M.; and Moritz, P. 2015 · 2015
Cited alongside, same era.
Concrete problems in AI safety
Original
Amodei, D.; Olah, C.; Steinhardt, J.; Christiano, P.; Schulman, J.; and Mané, D. 2016 · 2016
Cited alongside, same era.
Guided cost learning: Deep inverse optimal control via policy optimization
Finn, C.; Levine, S.; and Abbeel, P. 2016 · 2016
Cited alongside, same era.
Generative adversarial imitation learning
Ho, J.; and Ermon, S. 2016 · 2016
Cited alongside, same era.
End-to-end training of deep visuomotor policies
Levine, S.; Finn, C.; Darrell, T.; and Abbeel, P. 2016 · 2016
Cited alongside, same era.
Unsupervised representation learning with deep convolutional generative adversarial networks
Radford, A.; Metz, L.; and Chintala, S. 2016 · 2016
Cited alongside, same era.
Deep reinforcement learning from human preferences
Christiano, P. F.; Leike, J.; Brown, T.; Martic, M.; Legg, S.; and Amodei, D. 2017 · 2017
Cited alongside, same era.
Proximal policy optimization algorithms
Original
Schulman, J.; Wolski, F.; Dhariwal, P.; Radford, A.; and Klimov, O. 2017 · 2017
Cited alongside, same era.
Learning Robust Rewards with Adverserial Inverse Reinforcement Learning
Fu, J.; Luo, K.; and Levine, S. 2018 · 2018
Cited alongside, same era.

Then

Reward learning from human preferences and demonstrations in Atari
Ibarz, B.; Leike, J.; Pohlen, T.; Irving, G.; Legg, S.; and Amodei, D. 2018 · 2018
Later among the works it cites.
Illuminating Generalization in Deep Reinforcement Learning through Procedural Level Generation
Justesen, N.; Torrado, R.; Bontrager, P.; Khalifa, A.; Togelius, J.; and Risi, S. 2018 · 2018
Later among the works it cites.
Inverse reinforcement learning for video games
Tucker, A.; Gleave, A.; and Russell, S. 2018 · 2018
Later among the works it cites.
Minimalistic gridworld environment for openai gym
Chevalier-Boisvert, M.; Willems, L.; and Pal, S. 2019 · 2019
Later among the works it cites.
MineRL: a large-scale dataset of minecraft demonstrations
Guss, W. H.; Houghton, B.; Topin, N.; Wang, P.; Codel, C.; Veloso, M.; and Salakhutdinov, R. 2019 · 2019
Later among the works it cites.
Obstacle Tower: A Generalization Challenge in Vision, Control, and Planning
Juliani, A.; Khalifa, A.; Berges, V.-P.; Harper, J.; Teng, E.; Henry, H.; Crespi, A.; Togelius, J.; and Lange, D. 2019 · 2019
Later among the works it cites.
High-Fidelity Image Generation With Fewer Labels
Lučić, M.; Tschannen, M.; Ritter, M.; Zhai, X.; Bachem, O.; and Gelly, S. 2019 · 2019
Later among the works it cites.
SQIL: Imitation Learning via Reinforcement Learning with Sparse Rewards
Reddy, S.; Dragan, A. D.; and Levine, S. 2019 · 2019
Later among the works it cites.
Adversarial recovery of agent rewards from latent spaces of the limit order book
Roa-Vicens, J.; Wang, Y.; Mison, V.; Gal, Y.; and Silva, R. 2019 · 2019
Later among the works it cites.
DeepCrawl: Deep Reinforcement Learning for Turn Based Strategy Games
Sestini, A.; Kuhnle, A.; and Bagdanov, A. D. 2019 · 2019
Later among the works it cites.
Grandmaster level in StarCraft II using multi-agent reinforcement learning
Vinyals, O.; Babuschkin, I.; Czarnecki, W. M.; Mathieu, M.; Dudzik, A.; Chung, J.; Choi, D. H.; Powell, R.; Ewalds, T.; Georgiev, P.; et al. 2019 · 2019
Later among the works it cites.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…