2024

Mobility VLA: Multimodal Instruction Navigation with Long-Context VLMs and Topological Graphs

Chiang, Hao-Tien Lewis, Xu, Zhuo, Fu, Zipeng et al.

Understand

An elusive goal in navigation research is to build an intelligent agent that can understand multimodal instructions including natural language and image, and perform useful navigation.

To achieve this, we study a widely useful category of navigation tasks we call Multimodal Instruction Navigation with demonstration Tours (MINT), in which the environment prior is provided through a previously recorded demonstration video.
Recent advances in Vision Language Models (VLMs) have shown a promising path in achieving this goal as it demonstrates capabilities in perceiving and reasoning about multimodal inputs.
However, VLMs are typically trained to predict textual output and it is an open research question about how to best utilize them in navigation.

Built on

Analysis of probabilistic roadmaps for path planning
L. E. Kavraki, M. N. Kolountzakis, and J.-C. Latombe · 1998
Earlier work this paper cites.
Multiple view geometry in computer vision
R. Hartley and A. Zisserman · 2003
Earlier work this paper cites.
Using interpolation to improve path planning: The field d* algorithm
D. Ferguson and A. Stentz · 2006
Earlier work this paper cites.
Image recognition with an adiabatic quantum computer i. mapping to quadratic unconstrained binary optimization
Original
H. Neven, G. Rose, and W. G. Macready · 2008
Earlier work this paper cites.
The office marathon: Robust navigation in an indoor office environment
E. Marder-Eppstein, E. Berger, T. Foote, B. Gerkey, and K. Konolige · 2010
Earlier work this paper cites.
Structure-from-motion revisited
J. L. Schönberger and J.-M. Frahm · 2016
Earlier work this paper cites.
Pixelwise view selection for unstructured multi-view stereo
J. L. Schönberger, E. Zheng, M. Pollefeys, and J.-M. Frahm · 2016
Earlier work this paper cites.
Target-driven visual navigation in indoor scenes using deep reinforcement learning
Y. Zhu, R. Mottaghi, E. Kolve, J. J. Lim, A. Gupta, L. Fei-Fei, and A. Farhadi · 2017
Earlier work this paper cites.
Zero-shot deep reinforcement learning driving policy transfer for autonomous vehicles based on robust control
Z. Xu, C. Tang, and M. Tomizuka · 2018
Earlier work this paper cites.
On evaluation of embodied navigation agents
Original
P. Anderson, A. Chang, D. S. Chaplot, A. Dosovitskiy, S. Gupta, V. Koltun, J. Kosecka, J. Malik, R. Mottaghi, M. Savva, et al · 2018
Earlier work this paper cites.
Vision-and-language navigation: Interpreting visually-grounded navigation instructions in real environments
P. Anderson, Q. Wu, D. Teney, J. Bruce, M. Johnson, N. Sünderhauf, I. Reid, S. Gould, and A. Van Den Hengel · 2018
Earlier work this paper cites.
Semi-parametric topological memory for navigation
N. Savinov, A. Dosovitskiy, and V. Koltun · 2018
Earlier work this paper cites.
Disturbance-observer-based tracking controller for neural network driving policy transfer
C. Tang, Z. Xu, and M. Tomizuka · 2019
Earlier work this paper cites.
Toward modularization of neural network autonomous driving policy using parallel attribute networks
Z. Xu, H. Chang, C. Tang, C. Liu, and M. Tomizuka · 2019
Earlier work this paper cites.
Touchdown: Natural language navigation and spatial reasoning in visual street environments
H. Chen, A. Suhr, D. Misra, N. Snavely, and Y. Artzi · 2019
Earlier work this paper cites.
Visual representations for semantic target driven navigation
A. Mousavian, A. Toshev, M. Fišer, J. Košecká, A. Wahid, and J. Davidson · 2019
Earlier work this paper cites.
Scene memory transformer for embodied agents in long-horizon tasks
K. Fang, A. Toshev, L. Fei-Fei, and S. Savarese · 2019
Earlier work this paper cites.
Search on the replay buffer: Bridging planning and reinforcement learning
B. Eysenbach, R. R. Salakhutdinov, and S. Levine · 2019
Earlier work this paper cites.
Cascade attribute network: Decomposing reinforcement learning control policies using hierarchical neural networks
H. Chang, Z. Xu, and M. Tomizuka · 2020
Earlier work this paper cites.
Objectnav revisited: On evaluation of embodied agents navigating to objects
Original
D. Batra, A. Gokaslan, A. Kembhavi, O. Maksymets, R. Mottaghi, M. Savva, A. Toshev, and E. Wijmans · 2020
Earlier work this paper cites.
Object goal navigation using goal-oriented semantic exploration
D. S. Chaplot, D. P. Gandhi, A. Gupta, and R. R. Salakhutdinov · 2020
Earlier work this paper cites.

Similar

Room-across-room: Multilingual vision-and-language navigation with dense spatiotemporal grounding
Original
A. Ku, P. Anderson, R. Patel, E. Ie, and J. Baldridge · 2020
Cited alongside, same era.
Vision-and-dialog navigation
J. Thomason, M. Murray, M. Cakmak, and L. Zettlemoyer · 2020
Cited alongside, same era.
Learning 3d semantic scene graphs from 3d indoor reconstructions
J. Wald, H. Dhamo, N. Navab, and F. Tombari · 2020
Cited alongside, same era.
Semantic visual navigation by watching youtube videos
M. Chang, A. Gupta, and S. Gupta · 2020
Cited alongside, same era.
Unifying deep local and global features for image search
B. Cao, A. Araujo, and J. Sim · 2020
Cited alongside, same era.
trajax: differentiable optimal control on accelerators, 2021
R. Frostig, V. Sindhwani, S. Singh, and S. Tu · 2021
Cited alongside, same era.
Thda: Treasure hunt data augmentation for semantic navigation
O. Maksymets, V. Cartillier, A. Gokaslan, E. Wijmans, W. Galuba, S. Lee, and D. Batra · 2021
Cited alongside, same era.
Soat: A scene-and object-aware transformer for vision-and-language navigation
A. Moudgil, A. Majumdar, H. Agrawal, S. Lee, and D. Batra · 2021
Cited alongside, same era.
Representing scenes as neural radiance fields for view synthesis., 2021, 65
B. Mildenhall, P. Srinivasan, M. Tancik, J. Barron, R. Ramamoorthi, and R. N. Nerf · 2021
Cited alongside, same era.
Learning transferable visual models from natural language supervision
A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, et al · 2021
Cited alongside, same era.
Grouptron: Dynamic multi-scale graph convolutional networks for group-aware dense crowd trajectory forecasting
R. Zhou, H. Zhou, H. Gao, M. Tomizuka, J. Li, and Z. Xu · 2022
Cited alongside, same era.
Vision-and-language navigation: A survey of tasks, methods, and future directions
Original
J. Gu, E. Stefani, Q. Wu, J. Thomason, and X. E. Wang · 2022
Cited alongside, same era.

Then

Gemini: A family of highly capable multimodal models
Original
Gemini Team Google · 2023
Later among the works it cites.
Gpt-4 technical report
Original
J. Achiam, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, F. L. Aleman, D. Almeida, J. Altenschmidt, S. Altman, S. Anadkat, et al · 2023
Later among the works it cites.
Esc: Exploration with soft commonsense constraints for zero-shot object navigation
K. Zhou, K. Zheng, C. Pryor, Y. Shen, H. Jin, L. Getoor, and X. E. Wang · 2023
Later among the works it cites.
Nomad: Goal masked diffusion policies for navigation and exploration
Original
A. Sridhar, D. Shah, C. Glossop, and S. Levine · 2023
Later among the works it cites.
Offline visual representation learning for embodied navigation
K. Yadav, R. Ramrakhya, A. Majumdar, V.-P. Berges, S. Kuhar, D. Batra, A. Baevski, and O. Maksymets · 2023
Later among the works it cites.
Gnm: A general navigation model to drive any robot
D. Shah, A. Sridhar, A. Bhorkar, N. Hirose, and S. Levine · 2023
Later among the works it cites.
Palm 2 technical report
Original
R. Anil, A. M. Dai, O. Firat, M. Johnson, D. Lepikhin, A. Passos, S. Shakeri, E. Taropa, P. Bailey, Z. Chen, et al · 2023
Later among the works it cites.
Llama: Open and efficient foundation language models
Original
H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix, B. Rozière, N. Goyal, E. Hambro, F. Azhar, et al · 2023
Later among the works it cites.
Navigation with large language models: Semantic guesswork as a heuristic for planning
D. Shah, M. R. Equi, B. Osiński, F. Xia, B. Ichter, and S. Levine · 2023
Later among the works it cites.
Lm-nav: Robotic navigation with large pre-trained models of language, vision, and action
D. Shah, B. Osiński, S. Levine, et al · 2023
Later among the works it cites.
Gpt-driver: Learning to drive with gpt
Original
J. Mao, Y. Qian, H. Zhao, and Y. Wang · 2023
Later among the works it cites.
Cows on pasture: Baselines and benchmarks for language-driven zero-shot object navigation
S. Y. Gadre, M. Wortsman, G. Ilharco, L. Schmidt, and S. Song · 2023
Later among the works it cites.
Bridging zero-shot object navigation and foundation models through pixel-guided navigation skill
Original
W. Cai, S. Huang, G. Cheng, Y. Long, P. Gao, C. Sun, and H. Dong · 2023
Later among the works it cites.
Zip-nerf: Anti-aliased grid-based neural radiance fields
J. T. Barron, B. Mildenhall, D. Verbin, P. P. Srinivasan, and P. Hedman · 2023
Later among the works it cites.
Nerfmeshing: Distilling neural radiance fields into geometrically-accurate 3d meshes
M.-J. Rakotosaona, F. Manhardt, D. M. Arroyo, M. Niemeyer, A. Kundu, and F. Tombari · 2023
Later among the works it cites.
Matrix: Multi-agent trajectory generation with diverse contexts
Original
Z. Xu, R. Zhou, Y. Yin, H. Gao, M. Tomizuka, and J. Li · 2024
Closest in time.
Automate industrial inspection with anymal
ANYbotics · 2024
Closest in time.
Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context
Original
M. Reid, N. Savinov, D. Teplyashin, D. Lepikhin, T. Lillicrap, J.-b. Alayrac, R. Soricut, A. Lazaridou, O. Firat, J. Schrittwieser, et al · 2024
Closest in time.
Spatialvlm: Endowing vision-language models with spatial reasoning capabilities
Original
B. Chen, Z. Xu, S. Kirmani, B. Ichter, D. Driess, P. Florence, D. Sadigh, L. Guibas, and F. Xia · 2024
Closest in time.
Navgpt: Explicit reasoning in vision-and-language navigation with large language models
G. Zhou, Y. Hong, and Q. Wu · 2024
Closest in time.
Hello gpt-4o — openai, 2024
OpenAI · 2024
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…