2017

Trial without Error: Towards Safe Reinforcement Learning via Human Intervention

Saunders, William, Sastry, Girish, Stuhlmueller, Andreas et al.

Understand

AI systems are increasingly applied to complex tasks that involve interaction with humans.

During training, such systems are potentially dangerous, as they haven't yet learned to avoid actions that could cause serious harm.
How can an AI system explore and learn without making a single mistake that harms humans or otherwise causes serious damage? For model-free reinforcement learning, having a human "in the loop" and ready to intervene is currently the only way to prevent all catastrophes.
We formalize human intervention for RL and show how to reduce the human labor required by training a supervised learner to imitate the human's intervention decisions.

Built on

Active learning
Burr Settles · 2012
Earlier work this paper cites.
The arcade learning environment: An evaluation platform for general agents
Marc G Bellemare, Yavar Naddaf, Joel Veness, and Michael Bowling · 2013
Earlier work this paper cites.
Explaining and harnessing adversarial examples
Original
Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy · 2014
Earlier work this paper cites.
A Comprehensive Survey on Safe Reinforcement Learning
Javier Garcia and Fernando Fernandez · 2015
Earlier work this paper cites.
Safe exploration for optimization with gaussian processes
Yanan Sui, Alkis Gotovos, Joel Burdick, and Andreas Krause · 2015
Earlier work this paper cites.
Concrete problems in ai safety
Original
Dario Amodei, Chris Olah, Jacob Steinhardt, Paul Christiano, John Schulman, and Dan Mané · 2016
Earlier work this paper cites.
Unifying count-based exploration and intrinsic motivation
Marc Bellemare, Sriram Srinivasan, Georg Ostrovski, Tom Schaul, David Saxton, and Remi Munos · 2016
Earlier work this paper cites.
Model-free episodic control
Original
Charles Blundell, Benigno Uria, Alexander Pritzel, Yazhe Li, Avraham Ruderman, Joel Z Leibo, Jack Rae, Daan Wierstra, and Demis Hassabis · 2016
Earlier work this paper cites.
Openai gym
Original
Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba · 2016
Earlier work this paper cites.

Similar

Transfer from simulation to real world through learning deep inverse dynamics model
Original
Paul Christiano, Zain Shah, Igor Mordatch, Jonas Schneider, Trevor Blackwell, Joshua Tobin, Pieter Abbeel, and Wojciech Zaremba · 2016
Cited alongside, same era.
Deep learning for reward design to improve monte carlo tree search in atari games
Original
Xiaoxiao Guo, Satinder Singh, Richard Lewis, and Honglak Lee · 2016
Cited alongside, same era.
Deep learning of robotic tasks using strong and weak human supervision
Original
Bar Hilleli and Ran El-Yaniv · 2016
Cited alongside, same era.
Generative adversarial imitation learning
Jonathan Ho and Stefano Ermon · 2016
Cited alongside, same era.
Active reinforcement learning: Observing rewards at a cost
David Krueger, Jan Leike, Owain Evans, and John Salvatier · 2016
Cited alongside, same era.
Combating deep reinforcement learning’s sisyphean curse with reinforcement learning
Original
Zachary C Lipton, Abhishek Kumar, Jianfeng Gao, Lihong Li, and Li Deng · 2016
Cited alongside, same era.
Asynchronous methods for deep reinforcement learning
Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu · 2016
Cited alongside, same era.
Learning values across many orders of magnitude
Hado van Hasselt, Arthur Guez, Matteo Hessel, Volodymyr Mnih, and David Silver · 2016
Cited alongside, same era.
Deep reinforcement learning with double q-learning
Hado Van Hasselt, Arthur Guez, and David Silver · 2016
Cited alongside, same era.
Human-level control through deep reinforcement learning
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A. Rusu, Joel Veness, Marc G. Bellemare, Alex Graves, Martin Riedmiller, Andreas K. Fidjeland, Georg Ostrovski, Stig Petersen, Charles Beattie, Amir Sadik, Ioannis Antonoglou, Helen King, Dharshan Kumaran, Daan Wierstra, Shane Legg, and Demis Hassabis
Cited in the paper.
Human-level control through deep reinforcement learning
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al
Cited in the paper.

Then

Agent-agnostic human-in-the-loop reinforcement learning
Original
David Abel, John Salvatier, Andreas Stuhlmüller, and Owain Evans · 2017
Closest in time.
Background to “assessing russian activities and intentions in recent us elections”: The analytic process and cyber incident attribution
Intelligence Community Assessment · 2017
Closest in time.
Deep reinforcement learning from human preferences
Original
Paul Christiano, Jan Leike, Tom B Brown, Miljan Martic, Shane Legg, and Dario Amodei · 2017
Closest in time.
Offer: Off-environment reinforcement learning
Kamil Andrzej Ciosek and Shimon Whiteson · 2017
Closest in time.
Google waymo performing better than other self-driving cars, says california dmv
Maricris Francisco · 2017
Closest in time.
Openai baselines, a
OpenAI · 2017
Closest in time.
Openai universe starter agent, b
OpenAI · 2017
Closest in time.
Count-based exploration with neural density models
Original
Georg Ostrovski, Marc G. Bellemare, Aäron van den Oord, and Rémi Munos · 2017
Closest in time.
Tay (bot) — wikipedia, the free encyclopedia, 2017
Wikipedia · 2017
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…