Fetching the paper…

Reading the bibliography…

2025

OmniVLA: An Omni-Modal Vision-Language-Action Model for Robot Navigation

Hirose, Noriaki, Glossop, Catherine, Shah, Dhruv et al.

Original
PDF
Post on X

Understand

Humans can flexibly interpret and compose different goal specifications, such as language instructions, spatial coordinates, or visual references, when navigating to a destination.

In contrast, most existing robotic navigation policies are trained on a single modality, limiting their adaptability to real-world scenarios where different forms of goal specification are natural and complementary.
In this work, we present a training framework for robotic foundation models that enables omni-modal goal conditioning for vision-based navigation.
Our approach leverages a high-capacity vision-language-action (VLA) backbone and trains with three primary goal modalities: 2D poses, egocentric images, and natural language, as well as their combinations, through a randomized modality fusion strategy.

OmniVLA: An Omni-Modal Vision-Language-Action Model for Robot Navigation · Around

Built on

C. Lynch, M. Khansari, T. Xiao, V. Kumar, J. Tompson, S. Levine, and P. Sermanet, “Learning latent plans from play,”
Original
1903
Earlier work this paper cites.
H. Xu, Y. Gao, F. Yu, and T. Darrell, “End-to-end learning of driving models from large-scale video datasets,” in
2017
Earlier work this paper cites.
N. Savinov, A. Dosovitskiy, and V. Koltun, “Semi-parametric topological memory for navigation,”
Original
2018
Earlier work this paper cites.
G. Kahn, A. Villaflor, B. Ding, P. Abbeel, and S. Levine, “Self-supervised deep reinforcement learning with generalized computation graphs for robot navigation,” in
2018
Earlier work this paper cites.
N. Hirose, A. Sadeghian, M. Vázquez, P. Goebel, and S. Savarese, “Gonet: A semi-supervised deep learning approach for traversability estimation,” in
2018
Earlier work this paper cites.
N. Hirose, F. Xia, R. Martín-Martín, A. Sadeghian, and S. Savarese, “Deep visual mpc-policy learning for navigation,”
2019
Earlier work this paper cites.
D. Shah, B. Eysenbach, G. Kahn, N. Rhinehart, and S. Levine, “Rapid exploration for open-world navigation with latent goal models,”
Original
2021
Earlier work this paper cites.
N. Hirose, D. Shah, A. Sridhar, and S. Levine, “Exaug: Robot-conditioned navigation policies via geometric experience augmentation,”
Original
2022
Earlier work this paper cites.
D. Shah and S. Levine, “Viking: Vision-based kilometer-scale navigation with geographic hints,”
Original
2022
Earlier work this paper cites.
J. Gu, E. Stefani, Q. Wu, J. Thomason, and X. E. Wang, “Vision-and-language navigation: A survey of tasks, methods, and future directions,”
Original
2022
Earlier work this paper cites.
O. Mees, L. Hermann, and W. Burgard, “What matters in language conditioned robotic imitation learning over unstructured data,”
2022
Earlier work this paper cites.
S. Reed, K. Zolna, E. Parisotto, S. G. Colmenarejo, A. Novikov, G. Barth-Maron, M. Gimenez, Y. Sulsky, J. Kay, J. T. Springenberg,
Original
2022
Earlier work this paper cites.
T. Niwa, S. Taguchi, and N. Hirose, “Spatio-temporal graph localization networks for image-based navigation,” in
2022
Earlier work this paper cites.
M. Minderer, A. Gritsenko, A. Stone, M. Neumann, D. Weissenborn, A. Dosovitskiy, A. Mahendran, A. Arnab, M. Dehghani, Z. Shen,
2022
Earlier work this paper cites.
N. Hirose and K. Tahara, “Depth360: Self-supervised learning for monocular depth estimation using learnable camera distortion model,” in
2022
Earlier work this paper cites.

Similar

S. Triest, M. Sivaprakasam, S. J. Wang, W. Wang, A. M. Johnson, and S. Scherer, “Tartandrive: A large-scale dataset for learning off-road dynamics models,” in
2022
Cited alongside, same era.
A. Shaban, X. Meng, J. Lee, B. Boots, and D. Fox, “Semantic terrain classification for off-road autonomous driving,” in
2022
Cited alongside, same era.
H. Karnan, A. Nair, X. Xiao, G. Warnell, S. Pirk, A. Toshev, J. Hart, J. Biswas, and P. Stone, “Socially compliant navigation dataset (scand): A large-scale dataset of demonstrations for social navigation,”
2022
Cited alongside, same era.
D. Shah, A. Sridhar, A. Bhorkar, N. Hirose, and S. Levine, “Gnm: A general navigation model to drive any robot,” in
2023
Cited alongside, same era.
D. Shah, A. Sridhar, N. Dashora, K. Stachowicz, K. Black, N. Hirose, and S. Levine, “Vint: A foundation model for visual navigation,”
Original

Then

N. Hirose, C. Glossop, A. Sridhar, D. Shah, O. Mees, and S. Levine, “Lelan: Learning a language-conditioned navigation policy from in-the-wild videos,”
Original
2024
Later among the works it cites.
A.-C. Cheng, Y. Ji, Z. Yang, Z. Gongye, X. Zou, J. Kautz, E. Bıyık, H. Yin, S. Liu, and X. Wang, “Navila: Legged robot vision-language-action model for navigation,”
Original
2024
Later among the works it cites.
A. J. Sathyamoorthy, K. Weerakoon, M. Elnoor, A. Zore, B. Ichter, F. Xia, J. Tan, W. Yu, and D. Manocha, “Convoi: Context-aware navigation using vision language models in outdoor and indoor environments,” 2024. [Online]. Available:
Original
2024
Later among the works it cites.
Z. Xu, H.-T. L. Chiang, Z. Fu, M. G. Jacob, T. Zhang, T.-W. E. Lee, W. Yu, C. Schenck, D. Rendleman, D. Shah,
2024
Later among the works it cites.
O. M. Team, D. Ghosh, H. Walke, K. Pertsch, K. Black, O. Mees, S. Dasari, J. Hejna, T. Kreiman, C. Xu,

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

2023

Cited alongside, same era.

S. Y. Gadre, M. Wortsman, G. Ilharco, L. Schmidt, and S. Song, “Cows on pasture: Baselines and benchmarks for language-driven zero-shot object navigation,” in

2023

Cited alongside, same era.

D. Shah, B. Osiński, S. Levine,

2023

Cited alongside, same era.

D. Driess, F. Xia, M. S. Sajjadi, C. Lynch, A. Chowdhery, B. Ichter, A. Wahid, J. Tompson, Q. Vuong, T. Yu,

2023

Cited alongside, same era.

V. Myers, A. He, K. Fang, H. Walke, P. Hansen-Estruch, C.-A. Cheng, M. Jalobeanu, A. Kolobov, A. Dragan, and S. Levine, “Goal representations for instruction following: A semi-supervised language interface to control,” 2023

2023

Cited alongside, same era.

N. Hirose, D. Shah, A. Sridhar, and S. Levine, “Sacson: Scalable autonomous control for social navigation,”

2023

Cited alongside, same era.

J. Lin, H. Yin, W. Ping, P. Molchanov, M. Shoeybi, and S. Han, “Vila: On pre-training for visual language models,” in

2024

Cited alongside, same era.

M. Kim, K. Pertsch, S. Karamcheti, T. Xiao, A. Balakrishna, S. Nair, R. Rafailov, E. Foster, G. Lam, P. Sanketi, Q. Vuong, T. Kollar, B. Burchfiel, R. Tedrake, D. Sadigh, S. Levine, P. Liang, and C. Finn, “Openvla: An open-source vision-language-action model,”

2024

Cited alongside, same era.

2024

Later among the works it cites.

K. Black, N. Brown, D. Driess, A. Esmail, M. Equi, C. Finn, N. Fusai, L. Groom, K. Hausman, B. Ichter,

2024

Later among the works it cites.

R. Doshi, H. Walke, O. Mees, S. Dasari, and S. Levine, “Scaling cross-embodied learning: One policy for manipulation, navigation, locomotion and aviation,”

2024

Later among the works it cites.

FrodoBots, “Frodobots-2k,” 2024. [Online]. Available:

2024

Later among the works it cites.

S. Belkhale and D. Sadigh, “Minivla: A better vla with a smaller footprint,” 2024. [Online]. Available:

2024

Later among the works it cites.

M. Deitke, C. Clark, S. Lee, R. Tripathi, Y. Yang, J. S. Park, M. Salehi, N. Muennighoff, K. Lo, L. Soldaini,

2025

Closest in time.

N. Hirose, L. Ignatova, K. Stachowicz, C. Glossop, S. Levine, and D. Shah, “Learning to drive anywhere with model-based reannotation,”

2025

Closest in time.

C. Glossop, W. Chen, A. Bhorkar, D. Shah, and S. Levine, “Cast: Counterfactual labels improve instruction following in vision-language-action models,”

2025

Closest in time.

M. J. Kim, C. Finn, and P. Liang, “Fine-tuning vision-language-action models: Optimizing speed and success,”

2025

Closest in time.

“EarthRover Zero,”

2025

Closest in time.

M. Shukor, D. Aubakirova, F. Capuano, P. Kooijmans, S. Palma, A. Zouitine, M. Aractingi, C. Pascal, M. Russi, A. Marafioti,

2025

Closest in time.