2024

An Interactive Agent Foundation Model

Durante, Zane, Sarkar, Bidipta, Gong, Ran et al.

Understand

The development of artificial intelligence systems is transitioning from creating static, task-specific models to dynamic, agent-based systems capable of performing well in a wide range of applications.

We propose an Interactive Agent Foundation Model that uses a novel multi-task agent training paradigm for training AI agents across a wide range of domains, datasets, and tasks.
Our training paradigm unifies diverse pre-training strategies, including visual masked auto-encoders, language modeling, and next-action prediction, enabling a versatile and adaptable AI framework.
We demonstrate the performance of our framework across three separate domains -- Robotics, Gaming AI, and Healthcare.

Built on

The richmond agitation–sedation scale: validity and reliability in adult intensive care unit patients
Sessler, C. N., Gosnell, M. S., Grap, M. J., Brophy, G. M., O’Neal, P. V., Keane, K. A., Tesoro, E. P., and Elswick, R · 2002
Earlier work this paper cites.
Language conditioned imitation learning over unstructured data
Original
Lynch, C. and Sermanet, P · 2005
Earlier work this paper cites.
Reducing the risk of ventilator-acquired pneumonia through head of bed elevation
Keeley, L · 2007
Earlier work this paper cites.
Language models are few-shot learners
Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al · 2020
Earlier work this paper cites.
Retrieval augmented language model pre-training
Guu, K., Lee, K., Tung, Z., Pasupat, P., and Chang, M · 2020
Earlier work this paper cites.
Frozen in time: A joint video and image encoder for end-to-end retrieval
Bain, M., Nagrani, A., Varol, G., and Zisserman, A · 2021
Earlier work this paper cites.
On the opportunities and risks of foundation models
Original
Bommasani, R., Hudson, D. A., Adeli, E., Altman, R., Arora, S., von Arx, S., Bernstein, M. S., Bohg, J., Bosselut, A., Brunskill, E., et al · 2021
Earlier work this paper cites.
igibson 2.0: Object-centric simulation for robot learning of everyday household tasks
Original
Li, C., Xia, F., Martín-Martín, R., Lingelbach, M., Srivastava, S., Shen, B., Vainio, K., Gokmen, C., Dharan, G., Jain, T., et al · 2021
Earlier work this paper cites.
Learning transferable visual models from natural language supervision
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al · 2021
Earlier work this paper cites.
Minigpt-4: Enhancing vision-language understanding with advanced large language models, 2023
Zhu, D., Chen, J., Shen, X., Li, X., and Elhoseiny, M · 2021
Earlier work this paper cites.
Do as i can, not as i say: Grounding language in robotic affordances
Original
Ahn, M., Brohan, A., Brown, N., Chebotar, Y., Cortes, O., David, B., Finn, C., Gopalakrishnan, K., Hausman, K., Herzog, A., et al · 2022
Earlier work this paper cites.
Flamingo: a visual language model for few-shot learning
Alayrac, J.-B., Donahue, J., Luc, P., Miech, A., Barr, I., Hasson, Y., Lenc, K., Mensch, A., Millican, K., Reynolds, M., et al · 2022
Earlier work this paper cites.
Video pretraining (vpt): Learning to act by watching unlabeled online videos
Baker, B., Akkaya, I., Zhokov, P., Huizinga, J., Tang, J., Ecoffet, A., Houghton, B., Sampedro, R., and Clune, J · 2022
Earlier work this paper cites.
Rt-1: Robotics transformer for real-world control at scale
Original
Brohan, A., Brown, N., Carbajal, J., Chebotar, Y., Dabis, J., Finn, C., Gopalakrishnan, K., Hausman, K., Herzog, A., Hsu, J., et al · 2022
Earlier work this paper cites.
Clip-nav: Using clip for zero-shot vision-and-language navigation
Original
Dorbala, V. S., Sigurdsson, G., Piramuthu, R., Thomason, J., and Sukhatme, G. S · 2022
Earlier work this paper cites.
Masked autoencoders are scalable vision learners
He, K., Chen, X., Xie, S., Li, Y., Dollár, P., and Girshick, R · 2022
Earlier work this paper cites.
Vima: General robot manipulation with multimodal prompts
Jiang, Y., Gupta, A., Zhang, Z., Wang, G., Dou, Y., Chen, Y., Fei-Fei, L., Anandkumar, A., Zhu, Y., and Fan, L · 2022
Earlier work this paper cites.
Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation
Original
Li, J., Li, D., Xiong, C., and Hoi, S · 2022
Earlier work this paper cites.

Similar

Calvin: A benchmark for language-conditioned policy learning for long-horizon robot manipulation tasks
Mees, O., Hermann, L., Rosete-Beas, E., and Burgard, W · 2022
Cited alongside, same era.
Rethinking the role of demonstrations: What makes in-context learning work?
Original
Min, S., Lyu, X., Holtzman, A., Artetxe, M., Lewis, M., Hajishirzi, H., and Zettlemoyer, L · 2022
Cited alongside, same era.
A generalist agent
Reed, S., Zolna, K., Parisotto, E., Colmenarejo, S. G., Novikov, A., Barth-maron, G., Giménez, M., Sulsky, Y., Kay, J., Springenberg, J. T., et al · 2022
Cited alongside, same era.
Behavior: Benchmark for everyday household activities in virtual, interactive, and ecological environments
Srivastava, S., Li, C., Lingelbach, M., Martín-Martín, R., Xia, F., Vainio, K. E., Lian, Z., Gokmen, C., Buch, S., Liu, K., et al · 2022
Cited alongside, same era.
Self-instruct: Aligning language model with self generated instructions
Original
Wang, Y., Kordi, Y., Mishra, S., Liu, A., Smith, N. A., Khashabi, D., and Hajishirzi, H · 2022
Cited alongside, same era.
Opt: Open pre-trained transformer language models
Original
Zhang, S., Roller, S., Goyal, N., Artetxe, M., Chen, M., Chen, S., Dewan, C., Diab, M., Li, X., Lin, X. V., et al · 2022
Cited alongside, same era.
A review on innovation in healthcare sector (telehealth) through artificial intelligence
Amjad, A., Kordel, P., and Fernandes, G · 2023
Cited alongside, same era.
Rt-2: Vision-language-action models transfer web knowledge to robotic control
Original
Brohan, A., Brown, N., Carbajal, J., Chebotar, Y., Chen, X., Choromanski, K., Ding, T., Driess, D., Dubey, A., Finn, C., et al · 2023
Cited alongside, same era.
Bridging zero-shot object navigation and foundation models through pixel-guided navigation skill
Original
Cai, W., Huang, S., Cheng, G., Long, Y., Gao, P., Sun, C., and Dong, H · 2023
Cited alongside, same era.
Instructblip: Towards general-purpose vision-language models with instruction tuning, 2023
Dai, W., Li, J., Li, D., Tiong, A. M. H., Zhao, J., Wang, W., Li, B., Fung, P., and Hoi, S · 2023
Cited alongside, same era.
Can an embodied agent find your” cat-shaped mug”? llm-based zero-shot object navigation
Original
Dorbala, V. S., Mullen Jr, J. F., and Manocha, D · 2023
Cited alongside, same era.
Video language planning
Original
Du, Y., Yang, M., Florence, P., Xia, F., Wahid, A., Ichter, B., Sermanet, P., Yu, T., Abbeel, P., Tenenbaum, J. B., et al · 2023
Cited alongside, same era.

Then

Mo-vln: A multi-task benchmark for open-set zero-shot vision-and-language navigation
Original
Liang, X., Ma, L., Guo, S., Han, J., Xu, H., Ma, S., and Liang, X · 2023
Later among the works it cites.
Visual instruction tuning, 2023
Liu, H., Li, C., Wu, Q., and Lee, Y. J · 2023
Later among the works it cites.
Interactive language: Talking to robots in real time
Lynch, C., Wahid, A., Tompson, J., Ding, T., Betker, J., Baruch, R., Armstrong, T., and Florence, P · 2023
Later among the works it cites.
Eureka: Human-level reward design via coding large language models
Original
Ma, Y. J., Liang, W., Wang, G., Huang, D.-A., Bastani, O., Jayaraman, D., Zhu, Y., Fan, L., and Anandkumar, A · 2023
Later among the works it cites.
Multitask multimodal prompted training for interactive embodied task completion
Pantazopoulos, G., Nikandrou, M., Parekh, A., Hemanthage, B., Eshghi, A., Konstas, I., Rieser, V., Lemon, O., and Suglia, A · 2023
Later among the works it cites.
Human-assisted continual robot learning with foundation models
Original
Parakh, M., Fong, A., Simeonov, A., Gupta, A., Chen, T., and Agrawal, P · 2023
Later among the works it cites.
Check your facts and try again: Improving large language models with external knowledge and automated feedback
Original
Peng, B., Galley, M., He, P., Cheng, H., Xie, Y., Hu, Y., Huang, Q., Liden, L., Yu, Z., Chen, W., et al · 2023
Later among the works it cites.
Habitat 3.0: A co-habitat for humans, avatars and robots
Original
Puig, X., Undersander, E., Szot, A., Cote, M. D., Yang, T.-Y., Partsey, R., Desai, R., Clegg, A. W., Hlavac, M., Min, S. Y., et al · 2023
Later among the works it cites.
A survey of hallucination in large foundation models
Original
Rawte, V., Sheth, A., and Das, A · 2023
Later among the works it cites.
Stanford alpaca: An instruction-following llama model
Taori, R., Gulrajani, I., Zhang, T., Dubois, Y., Li, X., Guestrin, C., Liang, P., and Hashimoto, T. B · 2023
Later among the works it cites.
Llama: Open and efficient foundation language models
Original
Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., et al · 2023
Later among the works it cites.
Chatgpt empowered long-step robot control in various environments: A case application
Wake, N., Kanehira, A., Sasabuchi, K., Takamatsu, J., and Ikeuchi, K · 2023
Later among the works it cites.
Creative robot tool use with large language models
Original
Xu, M., Huang, P., Yu, W., Liu, S., Zhang, X., Niu, Y., Zhang, T., Xia, F., Tan, J., and Zhao, D · 2023
Later among the works it cites.
Language to rewards for robotic skill synthesis
Original
Yu, W., Gileadi, N., Fu, C., Kirmani, S., Lee, K.-H., Arenas, M. G., Chiang, H.-T. L., Erez, T., Hasenclever, L., Humplik, J., et al · 2023
Later among the works it cites.
Judging llm-as-a-judge with mt-bench and chatbot arena, 2023
Zheng, L., Chiang, W.-L., Sheng, Y., Zhuang, S., Wu, Z., Zhuang, Y., Lin, Z., Li, Z., Li, D., Xing, E. P., Zhang, H., Gonzalez, J. E., and Stoica, I · 2023
Later among the works it cites.
Navgpt: Explicit reasoning in vision-and-language navigation with large language models
Original
Zhou, G., Hong, Y., and Wu, Q · 2023
Later among the works it cites.
Agent ai: Surveying the horizons of multimodal interaction
Original
Durante, Z., Huang, Q., Wake, N., Gong, R., Park, J. S., Sarkar, B., Taori, R., Noda, Y., Terzopoulos, D., Choi, Y., et al · 2024
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…