2018

Inequity aversion improves cooperation in intertemporal social dilemmas

Hughes, Edward, Leibo, Joel Z., Phillips, Matthew G. et al.

Understand

Groups of humans are often able to find ways to cooperate with one another in complex, temporally extended social dilemmas.

Models based on behavioral economics are only able to explain this phenomenon for unrealistic stateless matrix games.
Recently, multi-agent reinforcement learning has been applied to generalize social dilemma problems to temporally and spatially extended Markov games.
However, this has not yet generated an agent that learns to cooperate in social dilemmas as humans do.

Built on

No. pt. 1 in Studi economico-giuridici pubblicati per cura della facoltà di Giurisprudenza della R. Università di Cagliari, Tipogr. di P. Cuppini, 1912
C. Gini, Variabilità e mutabilità: contributo allo studio delle distribuzioni e delle relazioni statistiche · 1912
Earlier work this paper cites.
D. Appleton-Century Company, 1938
B. F. Skinner, The Behavior of Organisms; An Experimental Analysis · 1938
Earlier work this paper cites.
G. R. Grice, “The relation of secondary reinforcement to delayed reward in visual discrimination learning.,” Journal of Experimental Psychology
1948
Earlier work this paper cites.
L. S. Shapley, “Stochastic Games,” In Proc. of the National Academy of Sciences of the United States of America
1953
Earlier work this paper cites.
H. L. A. Hart, “Are there any natural rights?,” The Philosophical Review
1955
Earlier work this paper cites.
J. Rawls, “Justice as fairness,” The philosophical review
1958
Earlier work this paper cites.
Harvard University Press, 1965
M. Olson, The logic of collective action · 1965
Earlier work this paper cites.
G. Hardin, “The tragedy of the commons,” Science
1968
Earlier work this paper cites.
T. C. Schelling, “Hockey helmets, concealed weapons, and daylight saving: A study of binary choices with externalities,” The Journal of Conflict Resolution
1973
Earlier work this paper cites.
P. Oliver, “Rewards and punishments as selective incentives for collective action: theoretical investigations,” American journal of sociology
1980
Earlier work this paper cites.
T. Yamagishi, “The provision of a sanctioning system as a public good,” vol. 51, pp. 110–116, 07 1986
1986
Earlier work this paper cites.
G. Klosko, “The principle of fairness and political obligation,” Ethics
1987
Earlier work this paper cites.
G. F. Loewenstein, L. Thompson, and M. H. Bazerman, “Social utility and decision making in interpersonal contexts.,” Journal of Personality and Social psychology
1989
Earlier work this paper cites.
Cambridge University Press, 1990
E. Ostrom, Governing the Commons: The Evolution of Institutions for Collective Action · 1990
Earlier work this paper cites.
E. Ostrom, J. Walker, and R. Gardner, “Covenants with and without a sword: Self-governance is possible.,” American political science Review
1992
Earlier work this paper cites.
Harvester Wheatsheaf, 1992
R. Gibbons, A primer in game theory · 1992
Earlier work this paper cites.
M. L. Littman, “Markov games as a framework for multi-agent reinforcement learning,” in Proceedings of the 11th International Conference on Machine Learning (ICML)
1994
Earlier work this paper cites.
B. S. Frey and I. Bohnet, “Institutions Affect Fairness: Experimental Investigations,” Journal of Institutional and Theoretical Economics
1995
Earlier work this paper cites.
T. W. Sandholm and R. H. Crites, “Multiagent reinforcement learning in the iterated prisoner’s dilemma,” Biosystems
1996
Earlier work this paper cites.
E. Ostrom, “A behavioral approach to the rational choice theory of collective action: Presidential address, american political science association, 1997,” American political science review
1998
Earlier work this paper cites.

Similar

P. Kollock, “Social dilemmas: The anatomy of cooperation,” Annual review of sociology
1998
Cited alongside, same era.
Cambridge, MA, USA: MIT Press, 1st ed., 1998
R. S. Sutton and A. G. Barto, Reinforcement Learning: An Introduction · 1998
Cited alongside, same era.
E. Fehr and K. M. Schmidt, “A theory of fairness, competition, and cooperation,” The quarterly journal of economics
1999
Cited alongside, same era.
M. J. Kearns and S. P. Singh, “Bias-variance error bounds for temporal difference updates,” in Proceedings of the Thirteenth Annual Conference on Computational Learning Theory
2000
Cited alongside, same era.
E. Fehr and S. Gachter, “Cooperation and punishment in public goods experiments,” American Economic Review
2000
Cited alongside, same era.
E. Fehr and S. Gächter, “Altruistic punishment in humans,” Nature
2002
Cited alongside, same era.
G. Charness and M. Rabin, “Understanding social preferences with simple tests,” The Quarterly Journal of Economics
2002
Cited alongside, same era.
W. E. Walsh, R. Das, G. Tesauro, and J. O. Kephart, “Analyzing complex strategic interactions in multi-agent systems,” in AAAI-02 Workshop on Game-Theoretic and Decision-Theoretic Agents
2002
Cited alongside, same era.
K. Verbeeck, J. Parent, and A. Nowé, “Homo egualis reinforcement learning agents for load balancing,” in Innovative Concepts for Agent-Based Systems, First International Workshop on Radical Agent Concepts, WRAC 2002, McLean, VA, USA, January 16-18, 2002, Revised Papers
2002
Cited alongside, same era.
T. Dietz, E. Ostrom, and P. C. Stern, “The struggle to govern the commons,” science
2003
Cited alongside, same era.
G. Chalkiadakis and C. Boutilier, “Coordination in multiagent reinforcement learning: a bayesian approach,” in The Second International Joint Conference on Autonomous Agents & Multiagent Systems, AAMAS 2003, July 14-18, 2003, Melbourne, Victoria, Australia, Proceedings
2003
Cited alongside, same era.
D. Engelmann and M. Strobel, “Inequality aversion, efficiency, and maximin preferences in simple distribution experiments,” American economic review
2004
Cited alongside, same era.

Then

M. A. Janssen, R. Holahan, A. Lee, and E. Ostrom, “Lab experiments for the study of social-ecological systems,” Science
2010
Later among the works it cites.
M. Janssen, “Introducing ecological dynamics into common-pool resource experiments,” Ecology and Society
2010
Later among the works it cites.
C. Bicchieri and A. Chavez, “Behaving as expected: Public information and fairness norms,” J. Behav. Decis. Making
2010
Later among the works it cites.
J. Henrich, J. Ensminger, R. McElreath, A. Barr, C. Barrett, et al
2010
Later among the works it cites.
C. Eckel and H. Gintis, “Blaming the messenger: Notes on the current state of experimental economics,” Journal of Economic Behavior & Organization
2010
Later among the works it cites.
S. de Jong and K. Tuyls, “Human-inspired computational fairness,” Autonomous Agents and Multi-Agent Systems
2011
Later among the works it cites.
D. G. Rand and M. A. Nowak, “Human cooperation,” Trends in cognitive sciences
2013
Later among the works it cites.
M. Janssen, “The role of information in governing the commons: experimental results,” Ecology and Society
2013
Later among the works it cites.
E. I. Hoppe and P. W. Schmitz, “Contracting under incomplete information and social preferences: An experimental study,” The Review of Economic Studies
2013
Later among the works it cites.
S. F. Brosnan and F. B. M. de Waal, “Evolution of responses to (un)fairness,” Science (New York, N.Y.)
2014
Later among the works it cites.
R. P. Blake, K. Mcauliffe, J. Corbit, T. Callaghan, O. Barry, A. Bowie, L. Kleutsch, K. Kramer, E. Ross, H. Vongsachang, R. Wrangham, and F. Warneken, “The ontogeny of fairness in seven societies,” vol. 528, 11 2015
2015
Later among the works it cites.
M. Kleiman-Weiner, M. K. Ho, J. L. Austerweil, M. L. Littman, and J. B. Tenenbaum, “Coordinate to cooperate or compete: Abstract goals and joint intentions in social interaction,” in CogSci
2016
Later among the works it cites.
V. Mnih, A. P. Badia, M. Mirza, A. Graves, T. P. Lillicrap, T. Harley, D. Silver, and K. Kavukcuoglu, “Asynchronous methods for deep reinforcement learning,” in Proceedings of the 33nd International Conference on Machine Learning, ICML 2016, New York City, NY, USA, June 19-24, 2016
2016
Later among the works it cites.
J. Z. Leibo, V. Zambaldi, M. Lanctot, J. Marecki, and T. Graepel, “Multi-agent Reinforcement Learning in Sequential Social Dilemmas,” in Proceedings of the 16th International Conference on Autonomous Agents and Multiagent Systems (AA-MAS 2017)
2017
Later among the works it cites.
J. Perolat, J. Z. Leibo, V. Zambaldi, C. Beattie, K. Tuyls, and T. Graepel, “A multi-agent reinforcement learning model of common-pool resource appropriation,” in Advances in Neural Information Processing Systems (NIPS)
2017
Later among the works it cites.
J. N. Foerster, R. Y. Chen, M. Al-Shedivat, S. Whiteson, P. Abbeel, and I. Mordatch, “Learning with opponent-learning awareness,” arXiv preprint arXiv:1709.04326
Original
2017
Later among the works it cites.
A. Lerer and A. Peysakhovich, “Maintaining cooperation in complex social dilemmas using deep reinforcement learning,” arXiv preprint arXiv:1707.01068
Original
2017
Later among the works it cites.
A. Peysakhovich and A. Lerer, “Prosocial learning agents solve generalized stag hunts better than selfish ones,” arXiv preprint arXiv:1709.02865
Original
2017
Later among the works it cites.
A. Peysakhovich and A. Lerer, “Consequentialist conditional cooperation in social dilemmas with imperfect information,” CoRR
Original
2017
Later among the works it cites.
P. Sunehag, G. Lever, A. Gruslys, W. M. Czarnecki, V. F. Zambaldi, M. Jaderberg, M. Lanctot, N. Sonnerat, J. Z. Leibo, K. Tuyls, and T. Graepel, “Value-decomposition networks for cooperative multi-agent learning,” CoRR
Original
2017
Later among the works it cites.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…