2023

$A^2$Nav: Action-Aware Zero-Shot Robot Navigation by Exploiting Vision-and-Language Ability of Foundation Models

Chen, Peihao, Sun, Xinyu, Zhi, Hongyan et al.

Understand

We study the task of zero-shot vision-and-language navigation (ZS-VLN), a practical yet challenging problem in which an agent learns to navigate following a path described by language instructions without requiring any path-instruction annotation data.

Normally, the instructions have complex grammatical structures and often contain various action descriptions (e.g., "proceed beyond", "depart from").
How to correctly understand and execute these action demands is a critical problem, and the absence of annotated data makes it even more challenging.
Note that a well-educated human being can easily understand path instructions without the need for any special training.

Built on

An rfid warehouse robot
Loh Poh Chuan, Ayob Johari, Mohd Helmy Abd Wahab, Danial Md Nor, Nik Shahidah Afifi Md Taujuddin, and Mohd Erdi Ayob · 2007
Earlier work this paper cites.
Automated robot picking system for e-commerce fulfillment warehouse application
C Liang, KJ Chee, Y Zou, H Zhu, A Causo, S Vidas, T Teng, IM Chen, KH Low, and CC Cheah · 2015
Earlier work this paper cites.
Vision-and-language navigation: Interpreting visually-grounded navigation instructions in real environments
Peter Anderson, Qi Wu, Damien Teney, Jake Bruce, Mark Johnson, Niko Sünderhauf, Ian D. Reid, Stephen Gould, and Anton van den Hengel · 2018
Earlier work this paper cites.
Vision-and-language navigation: Interpreting visually-grounded navigation instructions in real environments
Peter Anderson, Qi Wu, Damien Teney, Jake Bruce, Mark Johnson, Niko Sünderhauf, Ian D. Reid, Stephen Gould, and Anton van den Hengel · 2018
Earlier work this paper cites.
Speaker-follower models for vision-and-language navigation
Daniel Fried, Ronghang Hu, Volkan Cirik, Anna Rohrbach, Jacob Andreas, Louis-Philippe Morency, Taylor Berg-Kirkpatrick, Kate Saenko, Dan Klein, and Trevor Darrell · 2018
Earlier work this paper cites.
Look before you leap: Bridging model-free and model-based reinforcement learning for planned-ahead vision-and-language navigation
Xin Wang, Wenhan Xiong, Hongmin Wang, and William Yang Wang · 2018
Earlier work this paper cites.
A survey on the application trends of home service robotics
Georgios A. Zachiotis, George Andrikopoulos, Randy Gornez, Keisuke Nakamura, and George Nikolakopoulos · 2018
Earlier work this paper cites.
Visual landmark selection for generating grounded and interpretable navigation instructions
Sanyam Agarwal, Devi Parikh, Dhruv Batra, Peter Anderson, and Stefan Lee · 2019
Earlier work this paper cites.
BERT: pre-training of deep bidirectional transformers for language understanding
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova · 2019
Earlier work this paper cites.
Self-monitoring navigation agent via auxiliary progress estimation
Chih-Yao Ma, Jiasen Lu, Zuxuan Wu, Ghassan AlRegib, Zsolt Kira, Richard Socher, and Caiming Xiong · 2019
Earlier work this paper cites.
Habitat: A platform for embodied AI research
Manolis Savva, Jitendra Malik, Devi Parikh, Dhruv Batra, Abhishek Kadian, Oleksandr Maksymets, Yili Zhao, Erik Wijmans, Bhavana Jain, Julian Straub, Jia Liu, and Vladlen Koltun · 2019
Earlier work this paper cites.
Learning to navigate unseen environments: Back translation with environmental dropout
Hao Tan, Licheng Yu, and Mohit Bansal · 2019
Earlier work this paper cites.
Reinforced cross-modal matching and self-supervised imitation learning for vision-language navigation
Xin Wang, Qiuyuan Huang, Asli Celikyilmaz, Jianfeng Gao, Dinghan Shen, Yuan-Fang Wang, William Yang Wang, and Lei Zhang · 2019
Earlier work this paper cites.
Reinforced cross-modal matching and self-supervised imitation learning for vision-language navigation
Xin Wang, Qiuyuan Huang, Asli Celikyilmaz, Jianfeng Gao, Dinghan Shen, Yuan-Fang Wang, William Yang Wang, and Lei Zhang · 2019
Earlier work this paper cites.
Language models are few-shot learners
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei · 2020
Earlier work this paper cites.
Object goal navigation using goal-oriented semantic exploration
Devendra Singh Chaplot, Dhiraj Gandhi, Abhinav Gupta, and Ruslan Salakhutdinov · 2020
Earlier work this paper cites.
Threedworld: A platform for interactive multi-modal physical simulation
Original
Chuang Gan, Jeremy Schwartz, Seth Alter, Damian Mrowca, Martin Schrimpf, James Traer, Julian De Freitas, Jonas Kubilius, Abhishek Bhandwaldar, Nick Haber, et al · 2020
Earlier work this paper cites.
Towards learning a generic agent for vision-and-language navigation via pre-training
Weituo Hao, Chunyuan Li, Xiujun Li, Lawrence Carin, and Jianfeng Gao · 2020
Earlier work this paper cites.
Sub-instruction aware vision-and-language navigation
Yicong Hong, Cristian Rodriguez Opazo, Qi Wu, and Stephen Gould · 2020
Earlier work this paper cites.

Similar

Beyond the nav-graph: Vision-and-language navigation in continuous environments
Jacob Krantz, Erik Wijmans, Arjun Majumdar, Dhruv Batra, and Stefan Lee · 2020
Cited alongside, same era.
Beyond the nav-graph: Vision-and-language navigation in continuous environments
Jacob Krantz, Erik Wijmans, Arjun Majumdar, Dhruv Batra, and Stefan Lee · 2020
Cited alongside, same era.
Beyond the nav-graph: Vision-and-language navigation in continuous environments
Jacob Krantz, Erik Wijmans, Arjun Majumdar, Dhruv Batra, and Stefan Lee · 2020
Cited alongside, same era.
DD-PPO: learning near-perfect pointgoal navigators from 2.5 billion frames
Erik Wijmans, Abhishek Kadian, Ari Morcos, Stefan Lee, Irfan Essa, Devi Parikh, Manolis Savva, and Dhruv Batra · 2020
Cited alongside, same era.
Vision-language navigation with self-supervised auxiliary reasoning tasks
Fengda Zhu, Yi Zhu, Xiaojun Chang, and Xiaodan Liang · 2020
Cited alongside, same era.
Topological planning with transformers for vision-and-language navigation
Kevin Chen, Junshen K. Chen, Jo Chuang, Marynel Vázquez, and Silvio Savarese · 2021
Cited alongside, same era.
Grounding physical concepts of objects and events through dynamic visual reasoning
Zhenfang Chen, Jiayuan Mao, Jiajun Wu, Kwan-Yee Kenneth Wong, Joshua B Tenenbaum, and Chuang Gan · 2021
Cited alongside, same era.
Comphy: Compositional physical reasoning of objects and events from videos
Zhenfang Chen, Kexin Yi, Yunzhu Li, Mingyu Ding, Antonio Torralba, Joshua B Tenenbaum, and Chuang Gan · 2021
Cited alongside, same era.
VLN BERT: A recurrent vision-and-language BERT for navigation
Yicong Hong, Qi Wu, Yuankai Qi, Cristian Rodriguez Opazo, and Stephen Gould · 2021
Cited alongside, same era.
Waypoint models for instruction-guided navigation in continuous environments
Jacob Krantz, Aaron Gokaslan, Dhruv Batra, Stefan Lee, and Oleksandr Maksymets · 2021
Cited alongside, same era.
Waypoint models for instruction-guided navigation in continuous environments
Jacob Krantz, Aaron Gokaslan, Dhruv Batra, Stefan Lee, and Oleksandr Maksymets · 2021
Cited alongside, same era.
Audio-visual floorplan reconstruction
Senthil Purushwalkam, Sebastia Vicenc Amengual Gari, Vamsi Krishna Ithapu, Carl Schissler, Philip Robinson, Abhinav Gupta, and Kristen Grauman · 2021
Cited alongside, same era.

Then

Learning active camera for multi-object navigation
Peihao Chen, Dongyu Ji, Kunyang Lin, Weiwen Hu, Wenbing Huang, Thomas H. Li, Mingkui Tan, and Chuang Gan · 2022
Later among the works it cites.
Weakly-supervised multi-granularity map learning for vision-and-language navigation
Peihao Chen, Dongyu Ji, Kunyang Lin, Runhao Zeng, Thomas H. Li, Mingkui Tan, and Chuang Gan · 2022
Later among the works it cites.
Think global, act local: Dual-scale graph transformer for vision-and-language navigation
Shizhe Chen, Pierre-Louis Guhur, Makarand Tapaswi, Cordelia Schmid, and Ivan Laptev · 2022
Later among the works it cites.
Embodied concept learner: Self-supervised learning of concepts and mapping through instruction following
Mingyu Ding, Yan Xu, Zhenfang Chen, David Daniel Cox, Ping Luo, Joshua B Tenenbaum, and Chuang Gan · 2022
Later among the works it cites.
Clip-nav: Using CLIP for zero-shot vision-and-language navigation
Original
Vishnu Sashank Dorbala, Gunnar A. Sigurdsson, Robinson Piramuthu, Jesse Thomason, and Gaurav S. Sukhatme · 2022
Later among the works it cites.
CLIP on wheels: Zero-shot object navigation as object localization and exploration
Original
Samir Yitzhak Gadre, Mitchell Wortsman, Gabriel Ilharco, Ludwig Schmidt, and Shuran Song · 2022
Later among the works it cites.
The threedworld transport challenge: A visually guided task-and-motion planning benchmark towards physically realistic embodied ai
Chuang Gan, Siyuan Zhou, Jeremy Schwartz, Seth Alter, Abhishek Bhandwaldar, Dan Gutfreund, Daniel LK Yamins, James J DiCarlo, Josh McDermott, Antonio Torralba, et al · 2022
Later among the works it cites.
What can transformers learn in-context? a case study of simple function classes
Original
Shivam Garg, Dimitris Tsipras, Percy Liang, and Gregory Valiant · 2022
Later among the works it cites.
Cross-modal map learning for vision and language navigation
Georgios Georgakis, Karl Schmeckpeper, Karan Wanchoo, Soham Dan, Eleni Miltsakaki, Dan Roth, and Kostas Daniilidis · 2022
Later among the works it cites.
Language models as zero-shot planners: Extracting actionable knowledge for embodied agents
Wenlong Huang, Pieter Abbeel, Deepak Pathak, and Igor Mordatch · 2022
Later among the works it cites.
ZSON: zero-shot object-goal navigation using multimodal goal embeddings
Arjun Majumdar, Gunjan Aggarwal, Bhavika Devnani, Judy Hoffman, and Dhruv Batra · 2022
Later among the works it cites.
Object goal navigation with end-to-end self-supervision
Original
So Yeon Min, Yao-Hung Hubert Tsai, Wei Ding, Ali Farhadi, Ruslan Salakhutdinov, Yonatan Bisk, and Jian Zhang · 2022
Later among the works it cites.
PONI: potential functions for objectgoal navigation with interaction-free learning
Santhosh Kumar Ramakrishnan, Devendra Singh Chaplot, Ziad Al-Halah, Jitendra Malik, and Kristen Grauman · 2022
Later among the works it cites.
Lm-nav: Robotic navigation with large pre-trained models of language, vision, and action
Original
Dhruv Shah, Blazej Osinski, Brian Ichter, and Sergey Levine · 2022
Later among the works it cites.
Counterfactual cycle-consistent learning for instruction following and generation in vision-language navigation
Hanqing Wang, Wei Liang, Jianbing Shen, Luc Van Gool, and Wenguan Wang · 2022
Later among the works it cites.
Offline visual representation learning for embodied navigation
Original
Karmesh Yadav, Ram Ramrakhya, Arjun Majumdar, Vincent-Pierre Berges, Sachit Kuhar, Dhruv Batra, Alexei Baevski, and Oleksandr Maksymets · 2022
Later among the works it cites.
See, think, confirm: Interactive prompting between vision and language models for knowledge-based visual reasoning
Zhenfang Chen, Qinhong Zhou, Yikang Shen, Yining Hong, Hao Zhang, and Chuang Gan · 2023
Closest in time.
Building cooperative embodied agents modularly with large language models
Original
Hongxin Zhang, Weihua Du, Jiaming Shan, Qinhong Zhou, Yilun Du, Joshua B Tenenbaum, Tianmin Shu, and Chuang Gan · 2023
Closest in time.
Navgpt: Explicit reasoning in vision-and-language navigation with large language models
Original
Gengze Zhou, Yicong Hong, and Qi Wu · 2023
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…