2022

Instruction-Following Agents with Multimodal Transformer

Liu, Hao, Lee, Lisa, Lee, Kimin et al.

Understand

Humans are excellent at understanding language and vision to accomplish a wide range of tasks.

In contrast, creating general instruction-following embodied agents remains a difficult challenge.
Prior work that uses pure language-only models lack visual grounding, making it difficult to connect language instructions with visual observations.
On the other hand, methods that use pre-trained multimodal models typically come with divided language and visual representations, requiring designing specialized network architecture to fuse them together.

Built on

Adam: A method for stochastic optimization
Kingma, D. P. and Ba, J · 2015
Earlier work this paper cites.
U-net: Convolutional networks for biomedical image segmentation
Ronneberger, O., Fischer, P., and Brox, T · 2015
Earlier work this paper cites.
Aligning books and movies: Towards story-like visual explanations by watching movies and reading books
Zhu, Y., Kiros, R., Zemel, R., Salakhutdinov, R., Urtasun, R., Torralba, A., and Fidler, S · 2015
Earlier work this paper cites.
Yfcc100m: The new data in multimedia research
Thomee, B., Shamma, D. A., Friedland, G., Elizalde, B., Ni, K., Poland, D., Borth, D., and Li, L.-J · 2016
Earlier work this paper cites.
Decoupled weight decay regularization
Original
Loshchilov, I. and Hutter, F · 2017
Earlier work this paper cites.
Attention is all you need
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, Ł., and Polosukhin, I · 2017
Earlier work this paper cites.
Bert: Pre-training of deep bidirectional transformers for language understanding
Original
Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K · 2018
Earlier work this paper cites.
Film: Visual reasoning with a general conditioning layer
Perez, E., Strub, F., De Vries, H., Dumoulin, V., and Courville, A · 2018
Earlier work this paper cites.
Vilbert: Pretraining task-agnostic visiolinguistic representations for vision-and-language tasks
Lu, J., Batra, D., Parikh, D., and Lee, S · 2019
Earlier work this paper cites.
Multimodal transformer for unaligned multimodal language sequences
Tsai, Y.-H. H., Bai, S., Liang, P. P., Kolter, J. Z., Morency, L.-P., and Salakhutdinov, R · 2019
Earlier work this paper cites.
Language models are few-shot learners
Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al · 2020
Earlier work this paper cites.
An image is worth 16x16 words: Transformers for image recognition at scale
Original
Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., et al · 2020
Earlier work this paper cites.
Towards learning a generic agent for vision-and-language navigation via pre-training
Hao, W., Li, C., Li, X., Carin, L., and Gao, J · 2020
Earlier work this paper cites.

Similar

Human instruction-following with deep reinforcement learning via transfer-learning from text
Original
Hill, F., Mokra, S., Wong, N., and Harley, T · 2020
Cited alongside, same era.
Rlbench: The robot learning benchmark & learning environment
James, S., Ma, Z., Arrojo, D. R., and Davison, A. J · 2020
Cited alongside, same era.
Language conditioned imitation learning over unstructured data
Original
Lynch, C. and Sermanet, P · 2020
Cited alongside, same era.
Improving vision-and-language navigation with image-text pairs from the web
Majumdar, A., Shrivastava, A., Lee, S., Anderson, P., Parikh, D., and Batra, D · 2020
Cited alongside, same era.
Conceptual 12m: Pushing web-scale image-text pre-training to recognize long-tail visual concepts
Changpinyo, S., Sharma, P., Ding, N., and Soricut, R · 2021
Cited alongside, same era.
Redcaps: Web-curated image-text data created by the people, for the people
Desai, K., Kaul, G., Aysola, Z., and Johnson, J · 2021
Cited alongside, same era.
Airbert: In-domain pretraining for vision-and-language navigation
Guhur, P.-L., Tapaswi, M., Chen, S., Laptev, I., and Schmid, C · 2021
Cited alongside, same era.
What matters in learning from offline human demonstrations for robot manipulation
Original
Mandlekar, A., Xu, D., Wong, J., Nasiriany, S., Wang, C., Kulkarni, R., Fei-Fei, L., Savarese, S., Zhu, Y., and Martín-Martín, R · 2021
Cited alongside, same era.
Learning transferable visual models from natural language supervision
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al · 2021
Cited alongside, same era.
Creating multimodal interactive agents with imitation and self-supervised learning
Original
Team, D. I. A., Abramson, J., Ahuja, A., Brussee, A., Carnevale, F., Cassin, M., Fischer, F., Georgiev, P., Goldin, A., Harley, T., et al · 2021
Cited alongside, same era.
Do as i can, not as i say: Grounding language in robotic affordances
Original
Ahn, M., Brohan, A., Brown, N., Chebotar, Y., Cortes, O., David, B., Finn, C., Gopalakrishnan, K., Hausman, K., Herzog, A., et al · 2022
Cited alongside, same era.
Palm: Scaling language modeling with pathways
Original
Chowdhery, A., Narang, S., Devlin, J., Bosma, M., Mishra, G., Roberts, A., Barham, P., Chung, H. W., Sutton, C., Gehrmann, S., et al · 2022
Cited alongside, same era.

Then

Multimodal masked autoencoders learn transferable representations
Original
Geng, X., Liu, H., Lee, L., Schuurams, D., Levine, S., and Abbeel, P · 2022
Closest in time.
Instruction-driven history-aware policies for robotic manipulations
Original
Guhur, P.-L., Chen, S., Garcia, R., Tapaswi, M., Laptev, I., and Schmid, C · 2022
Closest in time.
Masked autoencoders are scalable vision learners
He, K., Chen, X., Xie, S., Li, Y., Dollár, P., and Girshick, R · 2022
Closest in time.
Language models as zero-shot planners: Extracting actionable knowledge for embodied agents
Original
Huang, W., Abbeel, P., Pathak, D., and Mordatch, I · 2022
Closest in time.
Q-attention: Enabling efficient learning for vision-based robotic manipulation
James, S. and Davison, A. J · 2022
Closest in time.
Bc-z: Zero-shot task generalization with robotic imitation learning
Jang, E., Irpan, A., Khansari, M., Kappler, D., Ebert, F., Lynch, C., Levine, S., and Finn, C · 2022
Closest in time.
Vima: General robot manipulation with multimodal prompts
Original
Jiang, Y., Gupta, A., Zhang, Z., Wang, G., Dou, Y., Chen, Y., Fei-Fei, L., Anandkumar, A., Zhu, Y., and Fan, L · 2022
Closest in time.
Simple but effective: Clip embeddings for embodied ai
Khandelwal, A., Weihs, L., Mottaghi, R., and Kembhavi, A · 2022
Closest in time.
Auto-lambda: Disentangling dynamic task relationships
Liu, S., James, S., Davison, A. J., and Johns, E · 2022
Closest in time.
A generalist agent
Original
Reed, S., Zolna, K., Parisotto, E., Colmenarejo, S. G., Novikov, A., Barth-Maron, G., Gimenez, M., Sulsky, Y., Kay, J., Springenberg, J. T., et al · 2022
Closest in time.
Behavior transformers: Cloning k k modes with one stone
Shafiullah, N. M. M., Cui, Z. J., Altanzaya, A., and Pinto, L · 2022
Closest in time.
Lm-nav: Robotic navigation with large pre-trained models of language, vision, and action
Original
Shah, D., Osinski, B., Ichter, B., and Levine, S · 2022
Closest in time.
Socratic models: Composing zero-shot multimodal reasoning with language
Original
Zeng, A., Wong, A., Welker, S., Choromanski, K., Tombari, F., Purohit, A., Ryoo, M., Sindhwani, V., Lee, J., Vanhoucke, V., et al · 2022
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…