Fetching the paper…

Reading the bibliography…

2024

Zero-shot Object Navigation with Vision-Language Models Reasoning

Wen, Congcong, Huang, Yisiyuan, Huang, Hao et al.

Original
PDF
Post on X

Understand

Object navigation is crucial for robots, but traditional methods require substantial training data and cannot be generalized to unknown environments.

Zero-shot object navigation (ZSON) aims to address this challenge, allowing robots to interact with unknown objects without specific training data.
Language-driven zero-shot object navigation (L-ZSON) is an extension of ZSON that incorporates natural language instructions to guide robot navigation and interaction with objects.
In this paper, we propose a novel Vision Language model with a Tree-of-thought Network (VLTNet) for L-ZSON.

Zero-shot Object Navigation with Vision-Language Models Reasoning · Around

Built on

Yamauchi, B.: A frontier-based approach for autonomous exploration. Proceedings IEEE International Symposium on Computational Intelligence in Robotics and Automation . https://doi.org/10.1109/cira.1997.613851
1997
Earlier work this paper cites.
Deitke, M., Han, W., Herrasti, A., Kembhavi, A., Kolve, E., Mottaghi, R., Salvador, J., Schwenk, D., VanderBilt, E., Wallingford, M., et al.: Robothor: An open simulation-to-real embodied ai platform (Apr 2020), https://arxiv.org/abs/2004.06799
Original
2004
Earlier work this paper cites.
Verbiest, K., Berrabah, S.A., Colon, E.: Autonomous frontier based exploration for mobile robots (Jan 2015), https://link.springer.com/chapter/10.1007/978-3-319-22873-0_1
2015
Earlier work this paper cites.
Bach, S.H., Broecheler, M., Huang, B., Getoor, L.: Hinge-loss markov random fields and probabilistic soft logic. Journal of Machine Learning Research 18
2017
Earlier work this paper cites.
He, K., Gkioxari, G., Dollar, P., Girshick, R.: Mask r-cnn. IEEE International Conference on Computer Vision (2017). https://doi.org/10.1109/iccv.2017.322
2017
Earlier work this paper cites.
Zhu, Y., Mottaghi, R., Kolve, E., Lim, J.J., Gupta, A., Fei-Fei, L., Farhadi, A.: Target-driven visual navigation in indoor scenes using deep reinforcement learning. IEEE International Conference on Robotics and Automation (2017). https://doi.org/10.1109/icra.2017.7989381
2017
Earlier work this paper cites.
Peters, M.E., Neumann, M., Iyyer, M., Gardner, M., Clark, C., Lee, K., Zettlemoyer, L.: Deep contextualized word representations (Mar 2018), https://arxiv.org/abs/1802.05365
Original
2018
Earlier work this paper cites.
Gomez, C., Hernandez, A.C., Barber, R.: Topological frontier-based exploration and map-building using semantic information (Oct 2019), https://www.mdpi.com/1424-8220/19/20/4595
2019
Earlier work this paper cites.
Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J.D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al.: Language models are few-shot learners (Jan 2020), https://proceedings.neurips.cc/paper_files/paper/2020/hash/1457c0d6bfcb4967418bfb8ac142f64a-Abstract.html
2020
Earlier work this paper cites.
Chang, M., Gupta, A., Gupta, S.: Semantic visual navigation by watching youtube videos (Jan 2020), https://proceedings.neurips.cc/paper/2020/hash/2cd4e8a2ce081c3d7c32c3cde4312ef7-Abstract.html
2020
Earlier work this paper cites.
Chattopadhyay, P., Hoffman, J., Mottaghi, R., Kembhavi, A.: Robustnav: Towards benchmarking robustness in embodied navigation (Jun 2021), https://arxiv.org/abs/2106.04531
Original
2021
Earlier work this paper cites.
Kamath, A., Singh, M., LeCun, Y., Synnaeve, G., Misra, I., Carion, N.: Mdetr - modulated detection for end-to-end multi-modal understanding. IEEE/CVF International Conference on Computer Vision (2021)
2021
Earlier work this paper cites.
Kim, W., Son, B., Kim, I.: Vilt: Vision-and-language transformer without convolution or region supervision (Jun 2021), https://arxiv.org/abs/2102.03334
Original
2021
Earlier work this paper cites.
Maksymets, O., Cartillier, V., Gokaslan, A., Wijmans, E., Galuba, W., Lee, S., Batra, D.: Thda: Treasure hunt data augmentation for semantic navigation. IEEE/CVF International Conference on Computer Vision (2021). https://doi.org/10.1109/iccv48922.2021.01509
2021
Earlier work this paper cites.

Similar

Radford, A., Kim, J.W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al.: Learning transferable visual models from natural language supervision (Feb 2021), https://arxiv.org/abs/2103.00020
Original
2021
Cited alongside, same era.
Ye, J., Batra, D., Das, A., Wijmans, E.: Auxiliary tasks and exploration enable objectgoal navigation. IEEE/CVF International Conference on Computer Vision (2021). https://doi.org/10.1109/iccv48922.2021.01581
2021
Cited alongside, same era.
Al-Halah, Z., Ramakrishnan, S.K., Grauman, K.: Zero experience required: Plug & play modular transfer learning for semantic visual navigation (Apr 2022), https://arxiv.org/abs/2202.02440
Original
2022
Cited alongside, same era.
Chen, P., Ji, D., Lin, K., Zeng, R., Li, T.H., Tan, M., Gan, C.: Weakly-supervised multi-granularity map learning for vision-and-language navigation (Oct 2022), https://arxiv.org/abs/2210.07506
Original
2022
Cited alongside, same era.

Then

Zheng, K., Zhou, K., Gu, J., Fan, Y., Wang, J., Di, Z., He, X., Wang, X.E.: Jarvis: A neuro-symbolic commonsense reasoning framework for conversational embodied agents (Sep 2022), https://arxiv.org/abs/2208.13266
Original
2022
Later among the works it cites.
Gadre, S.Y., Wortsman, M., Ilharco, G., Schmidt, L., Song, S.: Cows on pasture: Baselines and benchmarks for language-driven zero-shot object navigation. 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (2023). https://doi.org/10.1109/cvpr52729.2023.02219
2023
Later among the works it cites.
Huang, C., Mees, O., Zeng, A., Burgard, W.: Visual language maps for robot navigation. In: IEEE International Conference on Robotics and Automation. pp. 10608–10615. IEEE (2023)
2023
Later among the works it cites.
Leong, K.: Reinforcement learning with frontier-based exploration via autonomous environment (Jul 2023), https://arxiv.org/abs/2307.07296
Original
2023
Later among the works it cites.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Deitke, M., VanderBilt, E., Herrasti, A., Weihs, L., Ehsani, K., Salvador, J., Han, W., Kolve, E., Kembhavi, A., Mottaghi, R.: Procthor: Large-scale embodied ai using procedural generation (Dec 2022)

2022

Cited alongside, same era.

Dorbala, V.S., Sigurdsson, G., Piramuthu, R., Thomason, J., Sukhatme, G.S.: Clip-nav: Using clip for zero-shot vision-and-language navigation (Nov 2022), https://arxiv.org/abs/2211.16649

2022

Cited alongside, same era.

Gervet, T., Chintala, S., Batra, D., Malik, J., Chaplot, D.S.: Navigating to objects in the real world (Dec 2022), https://arxiv.org/abs/2212.00922

2022

Cited alongside, same era.

Khandelwal, A., Weihs, L., Mottaghi, R., Kembhavi, A.: Simple but effective: Clip embeddings for embodied ai. IEEE/CVF Conference on Computer Vision and Pattern Recognition (2022). https://doi.org/10.1109/cvpr52688.2022.01441

2022

Cited alongside, same era.

Li, L.H., Zhang, P., Zhang, H., Yang, J., Li, C., Zhong, Y., Wang, L., Yuan, L., Zhang, L., Hwang, J.N., et al.: Grounded language-image pre-training. IEEE/CVF Conference on Computer Vision and Pattern Recognition (2022). https://doi.org/10.1109/cvpr52688.2022.01069

2022

Cited alongside, same era.

Majumdar, A., Aggarwal, G., Devnani, B., Hoffman, J., Batra, D.: Zson: Zero-shot object-goal navigation using multimodal goal embeddings (Jun 2022)

2022

Cited alongside, same era.

Mezghan, L., Sukhbaatar, S., Lavril, T., Maksymets, O., Batra, D., Bojanowski, P., Alahari, K.: Memory-augmented reinforcement learning for image-goal navigation. IEEE/RSJ International Conference on Intelligent Robots and Systems (2022). https://doi.org/10.1109/iros47612.2022.9981090

2022

Cited alongside, same era.

Minderer, M., Gritsenko, A., Stone, A., Neumann, M., Weissenborn, D., Dosovitskiy, A., Mahendran, A., Arnab, A., Dehghani, M., Shen, Z., et al.: Simple open-vocabulary object detection with vision transformers (Jul 2022), https://arxiv.org/abs/2205.06230

2022

Cited alongside, same era.

Min, S.Y., Chaplot, D.S., Ravikumar, P.K., Bisk, Y., Salakhutdinov, R.: Film: Following instructions in language with modular methods (Sep 2023), https://openreview.net/forum?id=qI4542Y2s1D

2023

Later among the works it cites.

OpenAI: Gpt-3.5 technical report (2023)

2023

Later among the works it cites.

Park, J., Yoon, T., Hong, J., Yu, Y., Pan, M., Choi, S.: Zero-shot active visual search (zavis): Intelligent object search for robotic assistants. IEEE International Conference on Robotics and Automation (2023). https://doi.org/10.1109/icra48891.2023.10161345

2023

Later among the works it cites.

Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E., Le, Q., Zhou, D.: Chain-of-thought prompting elicits reasoning in large language models (Jan 2023), https://arxiv.org/abs/2201.11903

2023

Later among the works it cites.

Yao, S., Yu, D., Zhao, J., Shafran, I., Griffiths, T.L., Cao, Y., Narasimhan, K.: Tree of thoughts: Deliberate problem solving with large language models (May 2023), https://arxiv.org/abs/2305.10601

2023

Later among the works it cites.

Yu, B., Kasaei, H., Cao, M.: Frontier semantic exploration for visual target navigation. IEEE International Conference on Robotics and Automation (2023). https://doi.org/10.1109/icra48891.2023.10161059

2023

Later among the works it cites.

Yu, B., Kasaei, H., Cao, M.: L3mvn: Leveraging large language models for visual target navigation (Apr 2023), https://arxiv.org/abs/2304.05501

2023

Later among the works it cites.

Zhao, Q., Zhang, L., He, B., Qiao, H., Liu, Z.: Zero-shot object goal visual navigation. IEEE International Conference on Robotics and Automation (2023). https://doi.org/10.1109/icra48891.2023.10161289

2023

Later among the works it cites.

Zhou, K., Zheng, K., Pryor, C., Shen, Y., Jin, H., Getoor, L., Wang, X.E.: Esc: Exploration with soft commonsense constraints for zero-shot object navigation (Jul 2023), https://arxiv.org/abs/2301.13166

2023

Later among the works it cites.