2018

Penalizing side effects using stepwise relative reachability

Krakovna, Victoria, Orseau, Laurent, Kumar, Ramana et al.

Understand

How can we design safe reinforcement learning agents that avoid unnecessary disruptions to their environment? We show that current approaches to penalizing side effects can introduce bad incentives, e.g.

to prevent any irreversible changes in the environment, including the actions of other agents.
To isolate the source of such undesirable incentives, we break down side effects penalties into two components: a baseline state and a measure of deviation from this baseline state.
We argue that some of these incentives arise from the choice of baseline, and others arise from the choice of deviation measure.

Built on

Some philosophical problems from the standpoint of artificial intelligence
John McCarthy and Patrick J. Hayes · 1969
Earlier work this paper cites.
Algorithms for inverse reinforcement learning
Andrew Ng and Stuart Russell · 2000
Earlier work this paper cites.
Risk-sensitive reinforcement learning
Oliver Mihatsch and Ralph Neuneier · 2002
Earlier work this paper cites.
Apprenticeship learning via inverse reinforcement learning
Pieter Abbeel and Andrew Ng · 2004
Earlier work this paper cites.
All else being equal be empowered
Alexander S. Klyubin, Daniel Polani, and Chrystopher L. Nehaniv · 2005
Earlier work this paper cites.
A time-dependent hamilton-jacobi formulation of reachable sets for continuous dynamic games
Ian M. Mitchell, Alexandre M. Bayen, and Claire J. Tomlin · 2005
Earlier work this paper cites.
Maximum entropy inverse reinforcement learning
Brian D Ziebart, Andrew L Maas, J Andrew Bagnell, and Anind K Dey · 2008
Earlier work this paper cites.
Guaranteed safe online learning via reachability: tracking a ground target using a quadrotor
Jeremy H. Gillula and Claire J. Tomlin · 2012
Earlier work this paper cites.
Safe exploration in Markov decision processes
Teodor Mihai Moldovan and Pieter Abbeel · 2012
Earlier work this paper cites.
Safe exploration techniques for reinforcement learning — an overview
Martin Pecka and Tomas Svoboda · 2014
Earlier work this paper cites.
Empowerment — an introduction
Christoph Salge, Cornelius Glackin, and Daniel Polani · 2014
Earlier work this paper cites.
Risk-sensitive and robust decision-making: a CVaR optimization approach
Yinlam Chow, Aviv Tamar, Shie Mannor, and Marco Pavone · 2015
Earlier work this paper cites.

Similar

A comprehensive survey on safe reinforcement learning
Javier García and Fernando Fernández · 2015
Cited alongside, same era.
Variational information maximisation for intrinsically motivated reinforcement learning
Shakir Mohamed and Danilo J. Rezende · 2015
Cited alongside, same era.
Concrete problems in AI safety
Original
Dario Amodei, Chris Olah, Jacob Steinhardt, Paul Christiano, John Schulman, and Dan Mané · 2016
Cited alongside, same era.
Cooperative inverse reinforcement learning
Dylan Hadfield-Menell, Anca Dragan, Pieter Abbeel, and Stuart Russell · 2016
Cited alongside, same era.
Combating reinforcement learning’s sisyphean curse with intrinsic fear
Original
Zachary C. Lipton, Jianfeng Gao, Lihong Li, Jianshu Chen, and Li Deng · 2016
Cited alongside, same era.
Safely interruptible agents
Laurent Orseau and Stuart Armstrong · 2016
Cited alongside, same era.
Quantilizers: A safer alternative to maximizers for limited optimization
Jessica Taylor · 2016
Cited alongside, same era.
Alignment for advanced machine learning systems
Jessica Taylor, Eliezer Yudkowsky, Patrick LaVictoire, and Andrew Critch · 2016
Cited alongside, same era.
Safe exploration in finite Markov decision processes with Gaussian processes
Matteo Turchetta, Felix Berkenkamp, and Andreas Krause · 2016
Cited alongside, same era.
Low impact artificial intelligences
Original
Stuart Armstrong and Benjamin Levinstein · 2017
Cited alongside, same era.
Deep reinforcement learning from human preferences
Paul Christiano, Jan Leike, Tom B Brown, Miljan Martic, Shane Legg, and Dario Amodei · 2017
Cited alongside, same era.
Leave no Trace: Learning to reset for safe and autonomous reinforcement learning
Original
Benjamin Eysenbach, Shixiang Gu, Julian Ibarz, and Sergey Levine · 2017
Cited alongside, same era.

Then

A general safety framework for learning-based control in uncertain robotic systems
Original
Jaime F. Fisac, Anayo K. Akametalu, Melanie Nicole Zeilinger, Shahab Kaynama, Jeremy H. Gillula, and Claire J. Tomlin · 2017
Later among the works it cites.
Variational intrinsic control
Original
Karol Gregor, Danilo Jimenez Rezende, and Daan Wierstra · 2017
Later among the works it cites.
Inverse reward design
Dylan Hadfield-Menell, Smitha Milli, Pieter Abbeel, Stuart J. Russell, and Anca D. Dragan · 2017
Later among the works it cites.
AI safety gridworlds
Original
Jan Leike, Miljan Martic, Victoria Krakovna, Pedro A. Ortega, Tom Everitt, Andrew Lefrancq, Laurent Orseau, and Shane Legg · 2017
Later among the works it cites.
Trial without error: Towards safe reinforcement learning via human intervention
Original
William Saunders, Girish Sastry, Andreas Stuhlmueller, and Owain Evans · 2017
Later among the works it cites.
Safe exploration in continuous action spaces
Original
Gal Dalal, Krishnamurthy Dvijotham, Matej Vecerik, Todd Hester, Cosmin Paduraru, and Yuval Tassa · 2018
Closest in time.
Deep q-learning from demonstrations
Todd Hester, Matej Vecerik, Olivier Pietquin, Marc Lanctot, Tom Schaul, Bilal Piot, Dan Horgan, John Quan, Andrew Sendonaris, Ian Osband, Gabriel Dulac-Arnold, John Agapiou, Joel Z. Leibo, and Audrunas Gruslys · 2018
Closest in time.
Building safe artificial intelligence: specification, robustness, and assurance
Pedro Ortega, Vishal Maini, and et al · 2018
Closest in time.
Deep TAMER: interactive agent shaping in high-dimensional state spaces
Garrett Warnell, Nicholas R. Waytowich, Vernon Lawhern, and Peter Stone · 2018
Closest in time.
Minimax-regret querying on side effects for safe optimality in factored Markov decision processes
Shun Zhang, Edmund H. Durfee, and Satinder P. Singh · 2018
Closest in time.
Preferences implicit in the state of the world
Rohin Shah, Dmitrii Krasheninnikov, Jordan Alexander, Pieter Abbeel, and Anca Dragan · 2019
Closest in time.
Conservative agency via attainable utility preservation
Original
Alexander Turner, Dylan Hadfield-Menell, and Prasad Tadepalli · 2019
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…