Built on

Craig Boutilier, Thomas Dean, and Steve Hanks, Decision-theoretic planning: Structural assumptions and computational leverage , J. Artif. Int. Res. 11
1999
Earlier work this paper cites.
Stephen M Omohundro, The basic AI drives , AGI, vol. 171, 2008, pp. 483–492
2008
Earlier work this paper cites.
Judea Pearl, Causality , Cambridge university press, 2009
2009
Earlier work this paper cites.
Ross Shachter and David Heckerman, Pearl causality and the value of control , Heuristics, Probability, and Causality: A Tribute to Judea Pearl, College Publications, London, 2010, pp. 431–447
2010
Earlier work this paper cites.
Stuart Armstrong, Motivated value selection for artificial agents , Workshops at the Twenty-Ninth AAAI Conference on Artificial Intelligence, 2015
2015
Earlier work this paper cites.
Javier Garcıa and Fernando Fernández, A comprehensive survey on safe reinforcement learning , Journal of Machine Learning Research 16
2015
Earlier work this paper cites.
Nate Soares, Benja Fallenstein, Stuart Armstrong, and Eliezer Yudkowsky, Corrigibility , Workshops at the Twenty-Ninth AAAI Conference on Artificial Intelligence, 2015
2015
Earlier work this paper cites.
Dario Amodei, Chris Olah, Jacob Steinhardt, Paul Christiano, John Schulman, and Dan Mané, Concrete problems in AI safety , arXiv:1606.06565 (2016)
Original
2016
Earlier work this paper cites.

Similar

Laurent Orseau and Stuart Armstrong, Safely interruptible agents , Proceedings of the Thirty-Second Conference on Uncertainty in Artificial Intelligence, AUAI Press, 2016, pp. 557–566
2016
Cited alongside, same era.
Stuart Armstrong and Xavier O’Rourke, ‘Indifference’ methods for managing agent rewards , arXiv:1712.06365 (2017)
Original
2017
Cited alongside, same era.
Dylan Hadfield-Menell, Anca Dragan, Pieter Abbeel, and Stuart Russell, The off-switch game , Workshops at the Thirty-First AAAI Conference on Artificial Intelligence, 2017
2017
Cited alongside, same era.
Jan Leike, Miljan Martic, Victoria Krakovna, Pedro A Ortega, Tom Everitt, Andrew Lefrancq, Laurent Orseau, and Shane Legg, AI safety gridworlds , arXiv:1711.09883 (2017)
Original
2017
Cited alongside, same era.

Then

Tom Everitt and Marcus Hutter, Reward tampering problems and solutions in reinforcement learning: A causal influence diagram perspective , arXiv:1908.04734 (2019)
Original
2019
Later among the works it cites.
Tom Everitt, Ramana Kumar, Victoria Krakovna, and Shane Legg, Modeling AGI safety frameworks with causal influence diagrams , arXiv:1906.08663 (2019)
Original
2019
Later among the works it cites.
Koen Holtman, Corrigibility with utility preservation , arXiv:1908.01695 (2019)
Original
2019
Later among the works it cites.
Stuart Jonathan Russell, Human compatible: Artificial intelligence and the problem of control , Penguin Random House, 2019
2019
Later among the works it cites.
Ryan Carey, Eric Langlois, Tom Everitt, and Shane Legg, The incentives that shape behaviour , arXiv:2001.07118 (2020)

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…

2020

AGI Agent Safety by Iteratively Improving the Utility Function

Holtman, Koen

Understand

While it is still unclear if agents with Artificial General Intelligence (AGI) could ever be built, we can already use mathematical models to investigate potential safety systems for these agents.

We present an AGI safety layer that creates a special dedicated input terminal to support the iterative improvement of an AGI agent's utility function.
The humans who switched on the agent can use this terminal to close any loopholes that are discovered in the utility function's encoding of agent goals and constraints, to direct the agent towards new goals, or to force the agent to switch itself off.
An AGI agent may develop the emergent incentive to manipulate the above utility function improvement process, for example by deceiving, restraining, or even attacking the humans involved.

Built on

Craig Boutilier, Thomas Dean, and Steve Hanks, Decision-theoretic planning: Structural assumptions and computational leverage , J. Artif. Int. Res. 11
1999
Earlier work this paper cites.
Stephen M Omohundro, The basic AI drives , AGI, vol. 171, 2008, pp. 483–492
2008
Earlier work this paper cites.
Judea Pearl, Causality , Cambridge university press, 2009
2009
Earlier work this paper cites.
Ross Shachter and David Heckerman, Pearl causality and the value of control , Heuristics, Probability, and Causality: A Tribute to Judea Pearl, College Publications, London, 2010, pp. 431–447
2010
Earlier work this paper cites.
Stuart Armstrong, Motivated value selection for artificial agents , Workshops at the Twenty-Ninth AAAI Conference on Artificial Intelligence, 2015
2015
Earlier work this paper cites.
Javier Garcıa and Fernando Fernández, A comprehensive survey on safe reinforcement learning , Journal of Machine Learning Research 16
2015
Earlier work this paper cites.
Nate Soares, Benja Fallenstein, Stuart Armstrong, and Eliezer Yudkowsky, Corrigibility , Workshops at the Twenty-Ninth AAAI Conference on Artificial Intelligence, 2015
2015
Earlier work this paper cites.
Dario Amodei, Chris Olah, Jacob Steinhardt, Paul Christiano, John Schulman, and Dan Mané, Concrete problems in AI safety , arXiv:1606.06565 (2016)
Original
2016
Earlier work this paper cites.

Similar

Laurent Orseau and Stuart Armstrong, Safely interruptible agents , Proceedings of the Thirty-Second Conference on Uncertainty in Artificial Intelligence, AUAI Press, 2016, pp. 557–566
2016
Cited alongside, same era.
Stuart Armstrong and Xavier O’Rourke, ‘Indifference’ methods for managing agent rewards , arXiv:1712.06365 (2017)
Original
2017
Cited alongside, same era.
Dylan Hadfield-Menell, Anca Dragan, Pieter Abbeel, and Stuart Russell, The off-switch game , Workshops at the Thirty-First AAAI Conference on Artificial Intelligence, 2017
2017
Cited alongside, same era.
Jan Leike, Miljan Martic, Victoria Krakovna, Pedro A Ortega, Tom Everitt, Andrew Lefrancq, Laurent Orseau, and Shane Legg, AI safety gridworlds , arXiv:1711.09883 (2017)
Original
2017
Cited alongside, same era.

Then

Tom Everitt and Marcus Hutter, Reward tampering problems and solutions in reinforcement learning: A causal influence diagram perspective , arXiv:1908.04734 (2019)
Original
2019
Later among the works it cites.
Tom Everitt, Ramana Kumar, Victoria Krakovna, and Shane Legg, Modeling AGI safety frameworks with causal influence diagrams , arXiv:1906.08663 (2019)
Original
2019
Later among the works it cites.
Koen Holtman, Corrigibility with utility preservation , arXiv:1908.01695 (2019)
Original
2019
Later among the works it cites.
Stuart Jonathan Russell, Human compatible: Artificial intelligence and the problem of control , Penguin Random House, 2019
2019
Later among the works it cites.
Ryan Carey, Eric Langlois, Tom Everitt, and Shane Legg, The incentives that shape behaviour , arXiv:2001.07118 (2020)

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Tom Everitt, Gary Lea, and Marcus Hutter, AGI safety literature review , Proceedings of the 27th International Joint Conference on Artificial Intelligence, AAAI Press, 2018, pp. 5441–5449

2018

Cited alongside, same era.

Victoria Krakovna, Laurent Orseau, Ramana Kumar, Miljan Martic, and Shane Legg, Penalizing side effects using stepwise relative reachability , arXiv:1806.01186 (2018)

Original

2018

Cited alongside, same era.

Abram Demski and Scott Garrabrant, Embedded agency , arXiv:1902.09469 (2019)

Original

2019

Cited alongside, same era.

Original

2020

Closest in time.

Koen Holtman, Towards AGI agent safety by iteratively improving the utility function , Proceedings of the 13th International Conference on Artificial General Intelligence (AGI-20). Lecture Notes in Computer Science, vol 12177, Springer, 2020

2020

Closest in time.

Alexander Matt Turner, Dylan Hadfield-Menell, and Prasad Tadepalli, Conservative agency via attainable utility preservation , Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society, 2020, pp. 385–391

2020

Closest in time.