2021

Off-Belief Learning

Hu, Hengyuan, Lerer, Adam, Cui, Brandon et al.

Understand

The standard problem setting in Dec-POMDPs is self-play, where the goal is to find a set of policies that play optimally together.

Policies learned through self-play may adopt arbitrary conventions and implicitly rely on multi-step reasoning based on fragile assumptions about other agents' actions and thus fail when paired with humans or independently trained agents at test time.
To address this, we present off-belief learning (OBL).
At each timestep OBL agents follow a policy $\pi_1$ that is optimized assuming past actions were taken by a given, fixed policy ($\pi_0$), but assuming that future actions will be taken by $\pi_1$.

Built on

The strategy of conflict
Schelling, T. C · 1980
Earlier work this paper cites.
Multi-agent reinforcement learning: Independent vs. cooperative agents
Tan, M · 1993
Earlier work this paper cites.
The one-shot-deviation principle for sequential rationality
Hendon, E., Jacobsen, H. J., and Sloth, B · 1996
Earlier work this paper cites.
A cognitive hierarchy theory of one-shot games and experimental analysis
Camerer, C., Ho, T., and Chong, J.-K · 2003
Earlier work this paper cites.
Taming decentralized pomdps: Towards efficient policy computation for multiagent settings
Nair, R., Tambe, M., Yokoo, M., Pynadath, D., and Marsella, S · 2003
Earlier work this paper cites.
A cognitive hierarchy model of games
Camerer, C. F., Ho, T.-H., and Chong, J.-K · 2004
Earlier work this paper cites.
Ad hoc autonomous agent teams: Collaboration without pre-coordination
Stone, P., Kaminka, G. A., Kraus, S., and Rosenschein, J. S · 2010
Earlier work this paper cites.
Beliefs and endogenous cognitive levels: An experimental study
Agranov, M., Potamites, E., Schotter, A., and Tergiman, C · 2012
Earlier work this paper cites.
Predicting pragmatic reasoning in language games
Frank, M. C. and Goodman, N. D · 2012
Earlier work this paper cites.

Similar

Adam: A method for stochastic optimization
Original
Kingma, D. P. and Ba, J · 2015
Cited alongside, same era.
Human-level control through deep reinforcement learning
Mnih, V., Kavukcuoglu, K., Silver, D., Rusu, A. A., Veness, J., Bellemare, M. G., Graves, A., Riedmiller, M., Fidjeland, A. K., Ostrovski, G., Petersen, S., Beattie, C., Sadik, A., Antonoglou, I., King, H., Kumaran, D., Wierstra, D., Legg, S., and Hassabis, D · 2015
Cited alongside, same era.
Coordinate to cooperate or compete: abstract goals and joint intentions in social interaction
Kleiman-Weiner, M., Ho, M. K., Austerweil, J. L., Littman, M. L., and Tenenbaum, J. B · 2016
Cited alongside, same era.
Prioritized experience replay
Original
Schaul, T., Quan, J., Antonoglou, I., and Silver, D · 2016
Cited alongside, same era.
Deep reinforcement learning with double q-learning
van Hasselt, H., Guez, A., and Silver, D · 2016
Cited alongside, same era.
Dueling network architectures for deep reinforcement learning
Wang, Z., Schaul, T., Hessel, M., van Hasselt, H., Lanctot, M., and de Freitas, N · 2016
Cited alongside, same era.
Value-decomposition networks for cooperative multi-agent learning
Original
Sunehag, P., Lever, G., Gruslys, A., Czarnecki, W. M., Zambaldi, V. F., Jaderberg, M., Lanctot, M., Sonnerat, N., Leibo, J. Z., Tuyls, K., and Graepel, T · 2017
Cited alongside, same era.
Distributed prioritized experience replay
Horgan, D., Quan, J., Budden, D., Barth-Maron, G., Hessel, M., van Hasselt, H., and Silver, D · 2018
Cited alongside, same era.
QMIX: monotonic value function factorisation for deep multi-agent reinforcement learning
Rashid, T., Samvelyan, M., de Witt, C. S., Farquhar, G., Foerster, J. N., and Whiteson, S · 2018
Cited alongside, same era.

Then

On the utility of learning about humans for human-ai coordination
Carroll, M., Shah, R., Ho, M. K., Griffiths, T., Seshia, S. A., Abbeel, P., and Dragan, A. D · 2019
Later among the works it cites.
Multi-Agent Common Knowledge Reinforcement Learning
de Witt, C., Foerster, J., Farquhar, G., Torr, P., Boehmer, W., and Whiteson, S · 2019
Later among the works it cites.
Bayesian action decoder for deep multi-agent reinforcement learning
Foerster, J., Song, F., Hughes, E., Burch, N., Dunning, I., Whiteson, S., Botvinick, M., and Bowling, M · 2019
Later among the works it cites.
Recurrent experience replay in distributed reinforcement learning
Kapturowski, S., Ostrovski, G., Quan, J., Munos, R., and Dabney, W · 2019
Later among the works it cites.
The hanabi challenge: A new frontier for ai research
Bard, N., Foerster, J. N., Chandar, S., Burch, N., Lanctot, M., Song, H. F., Parisotto, E., Dumoulin, V., Moitra, S., Hughes, E., et al · 2020
Later among the works it cites.
Seed rl: Scalable and efficient deep-rl with accelerated central inference
Espeholt, L., Marinier, R., Stanczyk, P., Wang, K., and Michalski, M · 2020
Later among the works it cites.
Simplified action decoder for deep multi-agent reinforcement learning
Hu, H. and Foerster, J. N · 2020
Later among the works it cites.
“other-play”for zero-shot coordination
Hu, H., Peysakhovich, A., Lerer, A., and Foerster, J · 2020
Later among the works it cites.
Learned belief search: Efficiently improving policies in partially observable settings, 2021
Hu, H., Lerer, A., Brown, N., and Foerster, J · 2021
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…