Fetching the paper…

Reading the bibliography…

2023

Power-seeking can be probable and predictive for trained agents

Krakovna, Victoria, Kramar, Janos

Original
PDF
Post on X

Understand

Power-seeking behavior is a key source of risk from advanced AI, but our theoretical understanding of this phenomenon is relatively limited.

Building on existing theoretical results demonstrating power-seeking incentives for most reward functions, we investigate how the training process affects power-seeking incentives and show that they are still likely to hold for trained agents under some simplifying assumptions.
We formally define the training-compatible goal set (the set of goals consistent with the training rewards) and assume that the trained agent learns a goal from this set.
In a setting where the trained agent faces a choice to shut down or avoid shutdown in a new situation, we prove that the agent is likely to avoid shutdown.

Power-seeking can be probable and predictive for trained agents · Around

Built on

Optimal policies tend to seek power
Original
Alexander Matt Turner, Logan Smith, Rohin Shah, Andrew Critch, and Prasad Tadepalli · 1912
Earlier work this paper cites.
Is power-seeking AI an existential risk?
Original
Joseph Carlsmith · 2022
Earlier work this paper cites.
Without specific countermeasures, the easiest path to transformative AI likely leads to AI takeover
Ajeya Cotra · 2022
Earlier work this paper cites.

Similar

Goal misgeneralization in deep reinforcement learning
Original
Lauro Langosco, Jack Koch, Lee Sharkey, Jacob Pfau, Laurent Orseau, and David Krueger · 2022
Cited alongside, same era.
The alignment problem from a deep learning perspective
Original
Richard Ngo · 2022
Cited alongside, same era.
Goal misgeneralization: Why correct specifications aren’t enough for correct goals
Original
Rohin Shah, Vikrant Varma, Ramana Kumar, Mary Phuong, Victoria Krakovna, Jonathan Uesato, and Zac Kenton · 2022
Cited alongside, same era.

Then

Reward is not the optimization target
Alexander Matt Turner · 2022
Later among the works it cites.
Parametrically retargetable decision-makers tend to seek power
Original
Alexander Matt Turner and Prasad Tadepalli · 2022
Later among the works it cites.
Definitions of “objective" should be probable and predictive
Rohin Shah · 2023
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…