2023

Do Embodied Agents Dream of Pixelated Sheep: Embodied Decision Making using Language Guided World Modelling

Nottingham, Kolby, Ammanabrolu, Prithviraj, Suhr, Alane et al.

Understand

Reinforcement learning (RL) agents typically learn tabula rasa, without prior knowledge of the world.

However, if initialized with knowledge of high-level subgoals and transitions between subgoals, RL agents could utilize this Abstract World Model (AWM) for planning and exploration.
We propose using few-shot large language models (LLMs) to hypothesize an AWM, that will be verified through world experience, to improve sample efficiency of RL agents.
Our DECKARD agent applies LLM-guided exploration to item crafting in Minecraft in two phases: (1) the Dream phase where the agent uses an LLM to decompose a task into a sequence of subgoals, the hypothesized AWM; and (2) the Wake phase where the agent learns a modular policy for each subgoal and verifies or corrects the hypothesized AWM.

Built on

Learning to win by reading manuals in a monte-carlo framework
Branavan, S., Silver, D., and Barzilay, R · 2011
Earlier work this paper cites.
Proximal policy optimization algorithms
Original
Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and Klimov, O · 2017
Earlier work this paper cites.
Vision-and-language navigation: Interpreting visually-grounded navigation instructions in real environments
Anderson, P., Wu, Q., Teney, D., Bruce, J., Johnson, M., Sünderhauf, N., Reid, I., Gould, S., and van den Hengel, A · 2018
Earlier work this paper cites.
Embodied question answering
Das, A., Datta, S., Gkioxari, G., Lee, S., Parikh, D., and Batra, D · 2018
Earlier work this paper cites.
Iqa: Visual question answering in interactive environments
Gordon, D., Kembhavi, A., Rastegari, M., Redmon, J., Fox, D., and Farhadi, A · 2018
Earlier work this paper cites.
Interactive grounded language acquisition and generalization in a 2d world
Yu, H., Zhang, H., and Xu, W · 2018
Earlier work this paper cites.
BabyAI: First steps towards grounded language learning with a human in the loop
Chevalier-Boisvert, M., Bahdanau, D., Lahlou, S., Willems, L., Saharia, C., Nguyen, T. H., and Bengio, Y · 2019
Earlier work this paper cites.
Minerl: A large-scale dataset of minecraft demonstrations
Guss, W. H., Houghton, B., Topin, N., Wang, P., Codel, C., Veloso, M., and Salakhutdinov, R · 2019
Earlier work this paper cites.
Parameter-efficient transfer learning for nlp
Houlsby, N., Giurgiu, A., Jastrzebski, S., Morrone, B., De Laroussilhe, Q., Gesmundo, A., Attariyan, M., and Gelly, S · 2019
Earlier work this paper cites.
Language models as knowledge bases?
Petroni, F., Rocktäschel, T., Riedel, S., Lewis, P., Bakhtin, A., Wu, Y., and Miller, A · 2019
Earlier work this paper cites.
Composable modular reinforcement learning
Simpkins, C. and Isbell, C · 2019
Earlier work this paper cites.
How to avoid being eaten by a grue: Structured exploration strategies for textual worlds
Original
Ammanabrolu, P., Tien, E., Hausknecht, M., and Riedl, M. O · 2020
Earlier work this paper cites.
Experience grounds language
Bisk, Y., Holtzman, A., Thomason, J., Andreas, J., Bengio, Y., Chai, J., Lapata, M., Lazaridou, A., May, J., Nisnevich, A., et al · 2020
Earlier work this paper cites.
Language models are few-shot learners
Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al · 2020
Earlier work this paper cites.

Similar

Playing text-based games with common sense
Dambekodi, S., Frazier, S., Ammanabrolu, P., and Riedl, M · 2020
Cited alongside, same era.
Room-across-room: Multilingual vision-and-language navigation with dense spatiotemporal grounding
Ku, A., Anderson, P., Patel, R., Ie, E., and Baldridge, J · 2020
Cited alongside, same era.
Language conditioned imitation learning over unstructured data
Lynch, C. and Sermanet, P · 2020
Cited alongside, same era.
Align-rudder: Learning from few demonstrations by reward redistribution
Patil, V., Hofmarcher, M., Dinu, M.-C., Dorfer, M., Blies, P. M., Brandstetter, J., Arjona-Medina, J. A., and Hochreiter, S · 2020
Cited alongside, same era.
Alfred: A benchmark for interpreting grounded instructions for everyday tasks
Shridhar, M., Thomason, J., Gordon, D., Bisk, Y., Han, W., Mottaghi, R., Zettlemoyer, L., and Fox, D · 2020
Cited alongside, same era.
Rtfm: Generalising to new environment dynamics via reading
Zhong, V., Rocktäschel, T., and Grefenstette, E · 2020
Cited alongside, same era.
Learning knowledge graph-based world models of textual environments
Ammanabrolu, P. and Riedl, M · 2021
Cited alongside, same era.
Grounding language to entities and dynamics for generalization in reinforcement learning
Hanjie, A. W., Zhong, V. Y., and Narasimhan, K · 2021
Cited alongside, same era.
Compositional networks enable systematic generalization for grounded language understanding
Kuo, Y.-L., Katz, B., and Barbu, A · 2021
Cited alongside, same era.
Modular framework for visuomotor language grounding
Nottingham, K., Liang, L., Shin, D., Fowlkes, C. C., Fox, R., and Singh, S · 2021
Cited alongside, same era.
Forgetful experience replay in hierarchical reinforcement learning from expert demonstrations
Skrynnik, A., Staroverov, A., Aitygulov, E., Aksenov, K., Davydov, V., and Panov, A. I · 2021
Cited alongside, same era.
Embodied bert: A transformer model for embodied, language-guided visual task completion
Original
Suglia, A., Gao, Q., Thomason, J., Thattai, G., and Sukhatme, G · 2021
Cited alongside, same era.

Then

Video pretraining (vpt): Learning to act by watching unlabeled online videos
Baker, B., Akkaya, I., Zhokov, P., Huizinga, J., Tang, J., Ecoffet, A., Houghton, B., Sampedro, R., and Clune, J · 2022
Later among the works it cites.
A persistent spatial semantic representation for high-level natural language instruction execution
Blukis, V., Paxton, C., Fox, D., Garg, A., and Artzi, Y · 2022
Later among the works it cites.
Minedojo: Building open-ended embodied agents with internet-scale knowledge
Fan, L., Wang, G., Jiang, Y., Mandlekar, A., Yang, Y., Zhu, H., Tang, A., Huang, D.-A., Zhu, Y., and Anandkumar, A · 2022
Later among the works it cites.
Do as i can, not as i say: Grounding language in robotic affordances
Ichter, B., Brohan, A., Chebotar, Y., Finn, C., Hausman, K., Herzog, A., Ho, D., Ibarz, J., Irpan, A., Jang, E., Julian, R., Kalashnikov, D., Levine, S., Lu, Y., Parada, C., Rao, K., Sermanet, P., Toshev, A. T., Vanhoucke, V., Xia, F., Xiao, T., Xu, P., Yan, M., Brown, N., Ahn, M., Cortes, O., Sievers, N., Tan, C., Xu, S., Reyes, D., Rettinghouse, J., Quiambao, J., Pastor, P., Luu, L., Lee, K.-H., Kuang, Y., Jesmonth, S., Jeffrey, K., Ruano, R. J., Hsu, J., Gopalakrishnan, K., David, B., Zeng, A., and Fu, C. K · 2022
Later among the works it cites.
Transformer adapters for robot learning
Liang, A., Singh, I., Pertsch, K., and Thomason, J · 2022
Later among the works it cites.
Code as policies: Language model programs for embodied control
Liang, J., Huang, W., Xia, F., Xu, P., Hausman, K., Florence, P., Zeng, A., et al · 2022
Later among the works it cites.
Mind’s eye: Grounded language model reasoning through simulation
Original
Liu, R., Wei, J., Gu, S. S., Wu, T.-Y., Vosoughi, S., Cui, C., Zhou, D., and Dai, A. M · 2022
Later among the works it cites.
Improving intrinsic exploration with language abstractions
Mu, J., Zhong, V., Raileanu, R., Jiang, M., Goodman, N., Rocktäschel, T., and Grefenstette, E · 2022
Later among the works it cites.
Learning to query internet text for informing reinforcement learning agents
Nottingham, K., Pyla, A., Singh, S., and Fox, R · 2022
Later among the works it cites.
Powering next generation applications with openai codex, 2022
OpenAI · 2022
Later among the works it cites.
Progprompt: Generating situated robot task plans using large language models
Singh, I., Blukis, V., Mousavian, A., Goyal, A., Xu, D., Tremblay, J., Fox, D., Thomason, J., and Garg, A · 2022
Later among the works it cites.
Semantic exploration from language abstractions and pretrained representations
Original
Tam, A. C., Rabinowitz, N. C., Lampinen, A. K., Roy, N. A., Chan, S. C., Strouse, D., Wang, J. X., Banino, A., and Hill, F · 2022
Later among the works it cites.
Large language models still can’t plan (a benchmark for llms on planning and reasoning about change)
Valmeekam, K., Olmo, A., Sreedharan, S., and Kambhampati, S · 2022
Later among the works it cites.
Mastering diverse domains through world models, 2023
Original
Hafner, D., Pasukonis, J., Ba, J., and Lillicrap, T · 2023
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…