2021

Human irrationality: both bad and good for reward inference

Chan, Lawrence, Critch, Andrew, Dragan, Anca

Understand

Assuming humans are (approximately) rational enables robots to infer reward functions by observing human behavior.

But people exhibit a wide array of irrationalities, and our goal with this work is to better understand the effect they can have on reward inference.
The challenge with studying this effect is that there are many types of irrationality, with varying degrees of mathematical formalization.
We thus operationalize irrationality in the language of MDPs, by altering the Bellman optimality equation, and use this framework to study how these alterations would affect inference.

Built on

On the feasibility of learning, rather than assuming, human biases for reward inference
Original
Rohin Shah, Noah Gundotra, Pieter Abbeel, and Anca D Dragan · 1906
Earlier work this paper cites.
When is a linear control system optimal?
Rudolf Emil Kalman · 1964
Earlier work this paper cites.
Inverse problem of linear optimal control
Antony Jameson and Eliezer Kreindler · 1973
Earlier work this paper cites.
Illusions of control: How we overestimate our personal influence
Suzanne C Thompson · 1999
Earlier work this paper cites.
Algorithms for inverse reinforcement learning
Andrew Y Ng, Stuart J Russell, et al · 2000
Earlier work this paper cites.
Solving uncertain markov decision processes
J Andrew Bagnell, Andrew Y Ng, and Jeff G Schneider · 2001
Earlier work this paper cites.
Apprenticeship learning via inverse reinforcement learning
Pieter Abbeel and Andrew Y Ng · 2004
Earlier work this paper cites.
Bayesian inverse reinforcement learning
Deepak Ramachandran and Eyal Amir · 2007
Earlier work this paper cites.
Neural mechanisms mediating optimism bias
Tali Sharot, Alison M Riccardi, Candace M Raio, and Elizabeth A Phelps · 2007
Earlier work this paper cites.
Evaluations of pleasurable experiences: The peak-end rule
Amy M Do, Alexander V Rupert, and George Wolford · 2008
Earlier work this paper cites.
Maximum entropy inverse reinforcement learning
Brian D Ziebart, Andrew L Maas, J Andrew Bagnell, and Anind K Dey · 2008
Earlier work this paper cites.
Transfer learning for reinforcement learning domains: A survey
Matthew E Taylor and Peter Stone · 2009
Earlier work this paper cites.

Similar

Hyperbolically discounted temporal difference learning
William H Alexander and Joshua W Brown · 2010
Cited alongside, same era.
From human to humanoid locomotion—an inverse optimal control approach
Katja Mombaur, Anh Truong, and Jean-Paul Laumond · 2010
Cited alongside, same era.
Modeling interaction via the principle of maximum causal entropy
Brian D Ziebart, J Andrew Bagnell, and Anind K Dey · 2010
Cited alongside, same era.
Robust online optimization of reward-uncertain mdps
Kevin Regan and Craig Boutilier · 2011
Cited alongside, same era.
Continuous inverse optimal control with locally optimal examples
Original
Sergey Levine and Vladlen Koltun · 2012
Cited alongside, same era.
Legibility and predictability of robot motion
Anca D Dragan, Kenton CT Lee, and Siddhartha S Srinivasa · 2013
Cited alongside, same era.
Prospect theory: An analysis of decision under risk
Daniel Kahneman and Amos Tversky · 2013
Cited alongside, same era.
The Oxford Handbook of Cognitive Science
Susan E. F. Chipman · 2014
Cited alongside, same era.
Robots that can adapt like animals
Antoine Cully, Jeff Clune, Danesh Tarapore, and Jean-Baptiste Mouret · 2015
Cited alongside, same era.
Models of temporal discounting 1937–2000: An interdisciplinary exchange between economics and psychology
Till Grüne-Yanoff · 2015
Cited alongside, same era.
Openai gym.(2016)
Original
Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba · 2016
Cited alongside, same era.
Learning the preferences of ignorant, inconsistent agents
Owain Evans, Andreas Stuhlmüller, and Noah Goodman · 2016
Cited alongside, same era.

Then

Cooperative inverse reinforcement learning
Dylan Hadfield-Menell, Stuart J Russell, Pieter Abbeel, and Anca Dragan · 2016
Later among the works it cites.
Planning for autonomous cars that leverage effects on human actions
Dorsa Sadigh, Shankar Sastry, Sanjit A Seshia, and Anca D Dragan · 2016
Later among the works it cites.
An alternative softmax operator for reinforcement learning
Kavosh Asadi and Michael L Littman · 2017
Later among the works it cites.
Uncertain reward-transition mdps for negotiable reinforcement learning
Nishant Desai · 2017
Later among the works it cites.
Learning modular neural network policies for multi-task and multi-robot transfer
Coline Devin, Abhishek Gupta, Trevor Darrell, Pieter Abbeel, and Sergey Levine · 2017
Later among the works it cites.
Pragmatic-pedagogic value alignment
Original
Jaime F Fisac, Monica A Gates, Jessica B Hamrick, Chang Liu, Dylan Hadfield-Menell, Malayandi Palaniappan, Dhruv Malik, S Shankar Sastry, Thomas L Griffiths, and Anca D Dragan · 2017
Later among the works it cites.
Active preference-based learning of reward functions
Dorsa Sadigh, Anca D. Dragan, Shankar Sastry, and Sanjit A Seshia · 2017
Later among the works it cites.
Risk-sensitive inverse reinforcement learning via semi-and non-parametric methods
Original
Sumeet Singh, Jonathan Lacotte, Anirudha Majumdar, and Marco Pavone · 2017
Later among the works it cites.
Learning robust rewards with adverserial inverse reinforcement learning
Justin Fu, Katie Luo, and Sergey Levine · 2018
Later among the works it cites.
The surprising creativity of digital evolution: A collection of anecdotes from the evolutionary computation and artificial life research communities
Original
Joel Lehman, Jeff Clune, Dusan Misevic, Christoph Adami, Lee Altenberg, Julie Beaulieu, Peter J Bentley, Samuel Bernard, Guillaume Beslon, David M Bryson, et al · 2018
Later among the works it cites.
Where do you think you’re going?: Inferring beliefs about dynamics from behavior
Sid Reddy, Anca Dragan, and Sergey Levine · 2018
Later among the works it cites.
Task transfer by preference-based cost learning
Mingxuan Jing, Xiaojian Ma, Wenbing Huang, Fuchun Sun, and Huaping Liu · 2019
Later among the works it cites.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…