2023

Motif: Intrinsic Motivation from Artificial Intelligence Feedback

Klissarov, Martin, D'Oro, Pierluca, Sodhani, Shagun et al.

Understand

Exploring rich environments and evaluating one's actions without prior knowledge is immensely challenging.

In this paper, we propose Motif, a general method to interface such prior knowledge from a Large Language Model (LLM) with an agent.
Motif is based on the idea of grounding LLMs for decision-making without requiring them to interact with the environment: it elicits preferences from an LLM over pairs of captions to construct an intrinsic reward, which is then used to train agents with reinforcement learning.
We evaluate Motif's performance and behavior on the challenging, open-ended and procedurally-generated NetHack game.

Built on

Optimal Control of Markov Processes with Incomplete State Information I
Åström, Karl Johan · 1965
Earlier work this paper cites.
A possibility for implementing curiosity and boredom in model-building neural controllers
Jürgen Schmidhuber · 1991
Earlier work this paper cites.
Policy invariance under reward transformations: Theory and application to reward shaping
Andrew Y Ng, Daishi Harada, and Stuart Russell · 1999
Earlier work this paper cites.
Between MDPs and semi-MDPs: A framework for temporal abstraction in reinforcement learning
Richard S. Sutton, Doina Precup, and Satinder Singh · 1999
Earlier work this paper cites.
A social reinforcement learning agent
Charles Lee Isbell, Christian R. Shelton, Michael Kearns, Satinder Singh, and Peter Stone · 2001
Earlier work this paper cites.
Verification, the key to ai, 2001
Richard S. Sutton · 2001
Earlier work this paper cites.
The reward hypothesis, 2004
Richard S. Sutton · 2004
Earlier work this paper cites.
Intrinsic motivation systems for autonomous mental development
Pierre-Yves Oudeyer, Frdric Kaplan, and Verena V. Hafner · 2006
Earlier work this paper cites.
Reinforcement learning with human teachers: Evidence of feedback and guidance with implications for learning performance
Andrea Lockerd Thomaz, Cynthia Breazeal, et al · 2006
Earlier work this paper cites.
Coordinating with the future: The anticipatory nature of representation
Giovanni Pezzulo · 2008
Earlier work this paper cites.
Interactively shaping agents via human reinforcement: The tamer framework
W Bradley Knox and Peter Stone · 2009
Earlier work this paper cites.
Where do rewards come from?
Satinder Singh, R. Lewis, and A. Barto · 2009
Earlier work this paper cites.
Evolving a diversity of virtual creatures through novelty search and local competition
J. Lehman and K. O. Stanley · 2011
Earlier work this paper cites.
Preference-based reinforcement learning: a formal framework and a policy iteration algorithm
Johannes Fürnkranz, Eyke Hüllermeier, Weiwei Cheng, and et al · 2012
Earlier work this paper cites.
Robots that can adapt like animals
A. Cully, J. Clune, D. Tarapore, and J.-B. Mouret · 2015
Earlier work this paper cites.
Illuminating search spaces by mapping elites
Original
J. Mouret and J. Clune · 2015
Earlier work this paper cites.
Unifying count-based exploration and intrinsic motivation
Marc G. Bellemare, Sriram Srinivasan, Georg Ostrovski, Tom Schaul, David Saxton, and Rémi Munos · 2016
Earlier work this paper cites.
Deep reinforcement learning from human preferences
Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei · 2017
Earlier work this paper cites.
Quality and diversity optimization: A unifying modular framework
A. Cully and Y. Demiris · 2017
Earlier work this paper cites.
Value-Aware Loss Function for Model-based Reinforcement Learning
Amir-Massoud Farahmand, Andre Barreto, and Daniel Nikovski · 2017
Earlier work this paper cites.
Variational intrinsic control
Karol Gregor, Danilo J. Rezende, and Daan Wierstra · 2017
Earlier work this paper cites.
Interactive learning from policy-dependent human feedback
Original
James MacGlashan, Mark K. Ho, Robert Tyler Loftin, Bei Peng, Guan Wang, David L. Roberts, Matthew E. Taylor, and Michael L. Littman · 2017
Earlier work this paper cites.
A laplacian framework for option discovery in reinforcement learning
Marlos C. Machado, Marc G. Bellemare, and Michael Bowling · 2017
Earlier work this paper cites.
Improving policy gradient by exploring under-appreciated rewards
Ofir Nachum, Mohammad Norouzi, and Dale Schuurmans · 2017
Earlier work this paper cites.
Active preference-based learning of reward functions
Dorsa Sadigh, Anca D. Dragan, S. Shankar Sastry, and Sanjit A. Seshia · 2017
Earlier work this paper cites.
Proximal policy optimization algorithms
Original
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov · 2017
Earlier work this paper cites.
A survey of preference-based reinforcement learning methods
Christian Wirth, Riad Akrour, Gerhard Neumann, and Johannes Fürnkranz · 2017
Earlier work this paper cites.
Reinforcement learning: An introduction
Richard S Sutton and Andrew G Barto · 2018
Earlier work this paper cites.
Behavioral cloning from observation
Original
Faraz Torabi, Garrett Warnell, and Peter Stone · 2018
Earlier work this paper cites.
Learning to explore via meta-policy gradient
Tianbing Xu, Qiang Liu, Liang Zhao, and Jian Peng · 2018
Earlier work this paper cites.
Meta-gradient reinforcement learning
Zhongwen Xu, Hado van Hasselt, and David Silver · 2018
Earlier work this paper cites.
Deep reinforcement learning from policy-dependent human feedback
Original
Dilip Arumugam, Jun Ki Lee, Sophie Saskin, and Michael L. Littman · 2019
Earlier work this paper cites.

Similar

Large-scale study of curiosity-driven learning
Yuri Burda, Harrison Edwards, Deepak Pathak, Amos J. Storkey, Trevor Darrell, and Alexei A. Efros · 2019
Cited alongside, same era.
Language models are few-shot learners
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al · 2020
Cited alongside, same era.
Gradient-aware model-based policy search
Pierluca D’Oro, Alberto Maria Metelli, Andrea Tirinzoni, Matteo Papini, and Marcello Restelli · 2020
Cited alongside, same era.
Options of interest: Temporal abstraction with interest functions
Khimya Khetarpal, Martin Klissarov, Maxime Chevalier-Boisvert, Pierre-Luc Bacon, and Doina Precup · 2020
Cited alongside, same era.
The nethack learning environment
Heinrich Küttler, Nantas Nardelli, Alexander H. Miller, Roberta Raileanu, Marco Selvatici, Edward Grefenstette, and Tim Rocktäschel · 2020
Cited alongside, same era.
Offline reinforcement learning: Tutorial, review, and perspectives on open problems
Original
Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu · 2020
Cited alongside, same era.
Sample factory: Egocentric 3d control from pixels at 100000 FPS with asynchronous reinforcement learning
Aleksei Petrenko, Zhehui Huang, Tushar Kumar, Gaurav S. Sukhatme, and Vladlen Koltun · 2020
Cited alongside, same era.
Ride: Rewarding impact-driven exploration for procedurally-generated environments
Roberta Raileanu and Tim Rocktäschel · 2020
Cited alongside, same era.
On bonus based exploration methods in the arcade learning environment
Adrien Ali Taïga, William Fedus, Marlos C. Machado, Aaron C. Courville, and Marc G. Bellemare · 2020
Cited alongside, same era.
On the expressivity of markov reward
David Abel, William Dabney, Anna Harutyunyan, Michael K. Ho, Michael L. Littman, Doina Precup, and Satinder Singh · 2021
Cited alongside, same era.
Temporally-extended ϵ \epsilon -greedy exploration
W. Dabney, G. Ostrovski, and A. Barreto · 2021
Cited alongside, same era.
Meta dynamic programming
Pierluca D’Oro and Pierre-Luc Bacon · 2021
Cited alongside, same era.

Then

nle-sample-factory-baseline, 2022
Miffyli · 2022
Later among the works it cites.
Improving intrinsic exploration with language abstractions
Jesse Mu, Victor Zhong, Roberta Raileanu, Minqi Jiang, Noah D. Goodman, Tim Rocktäschel, and Edward Grefenstette · 2022
Later among the works it cites.
Training language models to follow instructions with human feedback
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al · 2022
Later among the works it cites.
A dataset perspective on offline reinforcement learning
Kajetan Schweighofer, Marius-constantin Dinu, Andreas Radler, Markus Hofmarcher, Vihang Prakash Patil, Angela Bitto-Nemling, Hamid Eghbal-zadeh, and Sepp Hochreiter · 2022
Later among the works it cites.
Defining and characterizing reward hacking
Original
Joar Skalse, Nikolaus H. R. Howe, Dmitrii Krasheninnikov, and David Krueger · 2022
Later among the works it cites.
Chain-of-thought prompting elicits reasoning in large language models
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al · 2022
Later among the works it cites.
React: Synergizing reasoning and acting in language models
Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik R Narasimhan, and Yuan Cao · 2022
Later among the works it cites.
Language reward modulation for pretraining reinforcement learning
Original
Ademi Adeniji, Amber Xie, Carmelo Sferrazza, Younggyo Seo, Stephen James, and Pieter Abbeel · 2023
Closest in time.
Learning about progress from experts
Jake Bruce, Ankit Anand, Bogdan Mazoure, and Rob Fergus · 2023
Closest in time.
Grounding large language models in interactive environments with online reinforcement learning, 2023
Thomas Carta, Clément Romac, Thomas Wolf, Sylvain Lamprier, Olivier Sigaud, and Pierre-Yves Oudeyer · 2023
Closest in time.
Mind2web: Towards a generalist agent for the web
Original
Xiang Deng, Yu Gu, Boyuan Zheng, Shijie Chen, Samuel Stevens, Boshi Wang, Huan Sun, and Yu Su · 2023
Closest in time.
Towards a unified agent with foundation models
Norman Di Palo, Arunkumar Byravan, Leonard Hasenclever, Markus Wulfmeier, Nicolas Heess, and Martin Riedmiller · 2023
Closest in time.
Guiding pretraining in reinforcement learning with large language models
Yuqing Du, Olivia Watkins, Zihan Wang, Cédric Colas, Trevor Darrell, Pieter Abbeel, Abhishek Gupta, and Jacob Andreas · 2023
Closest in time.
A study of global and episodic bonuses for exploration in contextual mdps
Mikael Henaff, Minqi Jiang, and Roberta Raileanu · 2023
Closest in time.
Language models can solve computer tasks
Original
Geunwoo Kim, Pierre Baldi, and Stephen McAleer · 2023
Closest in time.
Deep laplacian-based options for temporally-extended exploration
Martin Klissarov and Marlos C. Machado · 2023
Closest in time.
Efficient memory management for large language model serving with pagedattention
Original
Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E Gonzalez, Hao Zhang, and Ion Stoica · 2023
Closest in time.
Rlaif: Scaling reinforcement learning from human feedback with ai feedback, 2023
Harrison Lee, Samrat Phatale, Hassan Mansoor, Kellie Lu, Thomas Mesnard, Colton Bishop, Victor Carbune, and Abhinav Rastogi · 2023
Closest in time.
Steve-1: A generative model for text-to-behavior in minecraft, 2023
Shalev Lifshitz, Keiran Paster, Harris Chan, Jimmy Ba, and Sheila McIlraith · 2023
Closest in time.
LIV: language-image representations and rewards for robotic control
Yecheng Jason Ma, Vikash Kumar, Amy Zhang, Osbert Bastani, and Dinesh Jayaraman · 2023
Closest in time.
Mapl: Parameter-efficient adaptation of unimodal pre-trained models for vision-language few-shot prompting
Oscar Mañas, Pau Rodriguez Lopez, Saba Ahmadi, Aida Nematzadeh, Yash Goyal, and Aishwarya Agrawal · 2023
Closest in time.
Nethack is hard to hack
Original
Ulyana Piterbarg, Lerrel Pinto, and Rob Fergus · 2023
Closest in time.
Direct preference optimization: Your language model is secretly a reward model
Original
Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D Manning, and Chelsea Finn · 2023
Closest in time.
Policy optimization in a noisy neighborhood: On return landscapes in continuous control
Nathan Rahn, Pierluca D’Oro, Harley Wiltzer, Pierre-Luc Bacon, and Marc G Bellemare · 2023
Closest in time.
Proxy objectives in reinforcement learning from human feedback
John Schulman · 2023
Closest in time.
From pixels to ui actions: Learning to follow instructions via graphical user interfaces
Original
Peter Shaw, Mandar Joshi, James Cohan, Jonathan Berant, Panupong Pasupat, Hexiang Hu, Urvashi Khandelwal, Kenton Lee, and Kristina Toutanova · 2023
Closest in time.
Llama 2: Open foundation and fine-tuned chat models
Original
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al · 2023
Closest in time.
Voyager: An open-ended embodied agent with large language models
Original
Guanzhi Wang, Yuqi Xie, Yunfan Jiang, Ajay Mandlekar, Chaowei Xiao, Yuke Zhu, Linxi Fan, and Anima Anandkumar · 2023
Closest in time.
Spring: Gpt-4 out-performs rl algorithms by studying papers and reasoning
Original
Yue Wu, So Yeon Min, Shrimai Prabhumoye, Yonatan Bisk, Ruslan Salakhutdinov, Amos Azaria, Tom Mitchell, and Yuanzhi Li · 2023
Closest in time.
Language to rewards for robotic skill synthesis
Original
Wenhao Yu, Nimrod Gileadi, Chuyuan Fu, Sean Kirmani, Kuang-Huei Lee, Montse Gonzalez Arenas, Hao-Tien Lewis Chiang, Tom Erez, Leonard Hasenclever, Jan Humplik, et al · 2023
Closest in time.
Discovering policies with domino: Diversity optimization maintaining near optimality
Tom Zahavy, Yannick Schroecker, Feryal M. P. Behbahani, Kate Baumli, Sebastian Flennerhag, Shaobo Hou, and Satinder Singh · 2023
Closest in time.
Omni: Open-endedness via models of human notions of interestingness
Original
Jenny Zhang, Joel Lehman, Kenneth Stanley, and Jeff Clune · 2023
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…