Fetching the paper…

Reading the bibliography…

2024

Navi2Gaze: Leveraging Foundation Models for Navigation and Target Gazing

Zhu, Jun, Du, Zihao, Xu, Haotian et al.

Original
PDF
Post on X

Understand

Task-aware navigation continues to be a challenging area of research, especially in scenarios involving open vocabulary.

Previous studies primarily focus on finding suitable locations for task completion, often overlooking the importance of the robot's pose.
However, the robot's orientation is crucial for successfully completing tasks because of how objects are arranged (e.g., to open a refrigerator door).
Humans intuitively navigate to objects with the right orientation using semantics and common sense.

Navi2Gaze: Leveraging Foundation Models for Navigation and Target Gazing · Around

Built on

A. Saxena, J. Driemeyer, and A. Y. Ng, “Learning 3-d object orientation from images,” in 2009 IEEE International Conference on Robotics and Automation , 2009, pp. 794–800
2009
Earlier work this paper cites.
M. Sundermeyer, Z.-C. Marton, M. Durner, M. Brucker, and R. Triebel, “Implicit 3d orientation learning for 6d object detection from rgb images,” in The European Conference on Computer Vision (ECCV) , September 2018
2018
Earlier work this paper cites.
P. Anderson, A. Chang, D. S. Chaplot, A. Dosovitskiy, S. Gupta, V. Koltun, J. Kosecka, J. Malik, R. Mottaghi, M. Savva, et al. , “On evaluation of embodied navigation agents,” arXiv preprint arXiv:1807.06757 , 2018
Original
2018
Earlier work this paper cites.
M. Savva, A. Kadian, O. Maksymets, Y. Zhao, E. Wijmans, B. Jain, J. Straub, J. Liu, V. Koltun, J. Malik, D. Parikh, and D. Batra, “Habitat: A platform for embodied ai research,” in Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) , October 2019
2019
Earlier work this paper cites.
M. Kokic, D. Kragic, and J. Bohg, “Learning task-oriented grasping from human activity datasets,” IEEE Robotics and Automation Letters , vol. 5, no. 2, pp. 3352–3359, 2020
2020
Earlier work this paper cites.
T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, et al. , “Language models are few-shot learners,” Advances in neural information processing systems , vol. 33, pp. 1877–1901, 2020
2020
Earlier work this paper cites.
D. S. Chaplot, D. P. Gandhi, A. Gupta, and R. R. Salakhutdinov, “Object goal navigation using goal-oriented semantic exploration,” in Advances in Neural Information Processing Systems , H. Larochelle, M. Ranzato, R. Hadsell, M. Balcan, and H. Lin, Eds., vol. 33. Curran Associates, Inc., 2020, pp. 4247–4258. [Online]. Available: https://proceedings.neurips.cc/paper_files/paper/2020/file/2c75cf2681788adaca63aa95ae028b22-Paper.pdf
2020
Earlier work this paper cites.
A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, G. Krueger, and I. Sutskever, “Learning transferable visual models from natural language supervision,” in Proceedings of the 38th International Conference on Machine Learning , ser. Proceedings of Machine Learning Research, M. Meila and T. Zhang, Eds., vol. 139. PMLR, 18–24 Jul 2021, pp. 8748–8763
2021
Earlier work this paper cites.
X. Gu, T.-Y. Lin, W. Kuo, and Y. Cui, “Open-vocabulary object detection via vision and language knowledge distillation,” arXiv preprint arXiv:2104.13921 , 2021
Original
2021
Earlier work this paper cites.
S. K. Ramakrishnan, A. Gokaslan, E. Wijmans, O. Maksymets, A. Clegg, J. M. Turner, E. Undersander, W. Galuba, A. Westbury, A. X. Chang, M. Savva, Y. Zhao, and D. Batra, “Habitat-matterport 3d dataset (HM3d): 1000 large-scale 3d environments for embodied AI,” in Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2) , 2021. [Online]. Available: https://openreview.net/forum?id=-v4OuqNs5P
2021
Earlier work this paper cites.
K. Wada, S. James, and A. J. Davison, “Reorientbot: Learning object reorientation for specific-posed placement,” in 2022 International Conference on Robotics and Automation (ICRA) , 2022, pp. 8252–8258
2022
Earlier work this paper cites.
A. Chowdhery, S. Narang, J. Devlin, M. Bosma, G. Mishra, A. Roberts, P. Barham, H. W. Chung, C. Sutton, S. Gehrmann, et al. , “Palm: Scaling language modeling with pathways,” arXiv preprint arXiv:2204.02311 , 2022
Original
2022
Earlier work this paper cites.
A. Majumdar, G. Aggarwal, B. Devnani, J. Hoffman, and D. Batra, “Zson: Zero-shot object-goal navigation using multimodal goal embeddings,” in Advances in Neural Information Processing Systems , S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, Eds., vol. 35. Curran Associates, Inc., 2022, pp. 32 340–32 352. [Online]. Available: https://proceedings.neurips.cc/paper_files/paper/2022/file/d0b8f0c8f79d3a621af945cafb669f4b-Paper-Conference.pdf
2022
Earlier work this paper cites.
L. H. Li, P. Zhang, H. Zhang, J. Yang, C. Li, Y. Zhong, L. Wang, L. Yuan, L. Zhang, J.-N. Hwang, K.-W. Chang, and J. Gao, “Grounded language-image pre-training,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , June 2022, pp. 10 965–10 975
2022
Earlier work this paper cites.
B. Li, K. Q. Weinberger, S. Belongie, V. Koltun, and R. Ranftl, “Language-driven semantic segmentation,” arXiv preprint arXiv:2201.03546 , 2022
Original
2022
Earlier work this paper cites.

Similar

S. Subramanian, W. Merrill, T. Darrell, M. Gardner, S. Singh, and A. Rohrbach, “Reclip: A strong zero-shot baseline for referring expression comprehension,” arXiv preprint arXiv:2204.05991 , 2022
Original
2022
Cited alongside, same era.
K. Yadav, S. K. Ramakrishnan, J. Turner, A. Gokaslan, O. Maksymets, R. Jain, R. Ramrakhya, A. X. Chang, A. Clegg, M. Savva, E. Undersander, D. S. Chaplot, and D. Batra, “Habitat challenge 2022,” https://aihabitat.org/challenge/2022/ , 2022
2022
Cited alongside, same era.
S. Yenamandra, A. Ramachandran, K. Yadav, A. S. Wang, M. Khanna, T. Gervet, T.-Y. Yang, V. Jain, A. Clegg, J. M. Turner, Z. Kira, M. Savva, A. X. Chang, D. S. Chaplot, D. Batra, R. Mottaghi, Y. Bisk, and C. Paxton, “Homerobot: Open-vocabulary mobile manipulation,” in 7th Annual Conference on Robot Learning , 2023. [Online]. Available: https://openreview.net/forum?id=b-cto-fetlz
2023
Cited alongside, same era.

Then

H. Dai, C. Ma, Z. Liu, Y. Li, P. Shu, X. Wei, L. Zhao, Z. Wu, D. Zhu, W. Liu, et al. , “Samaug: Point prompt augmentation for segment anything model,” arXiv preprint arXiv:2307.01187 , 2023
Original
2023
Later among the works it cites.
L. Tang, H. Xiao, and B. Li, “Can sam segment anything? when sam meets camouflaged object detection,” arXiv preprint arXiv:2304.04709 , 2023
Original
2023
Later among the works it cites.
R. Zhang, Z. Jiang, Z. Guo, S. Yan, J. Pan, X. Ma, H. Dong, P. Gao, and H. Li, “Personalize segment anything model with one shot,” arXiv preprint arXiv:2305.03048 , 2023
Original
2023
Later among the works it cites.
F. Rajič, L. Ke, Y.-W. Tai, C.-K. Tang, M. Danelljan, and F. Yu, “Segment anything meets point tracking,” arXiv preprint arXiv:2307.01197 , 2023
Original
2023
Later among the works it cites.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Z. He, N. Chavan-Dafle, J. Huh, S. Song, and V. Isler, “Pick2place: Task-aware 6dof grasp estimation via object-centric perspective affordance,” in 2023 IEEE International Conference on Robotics and Automation (ICRA) . IEEE, 2023, pp. 7996–8002

2023

Cited alongside, same era.

Y. Hai, R. Song, J. Li, and Y. Hu, “Shape-constraint recurrent flow for 6d object pose estimation,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2023, pp. 4831–4840

2023

Cited alongside, same era.

B. Wen, J. Tremblay, V. Blukis, S. Tyree, T. Müller, A. Evans, D. Fox, J. Kautz, and S. Birchfield, “Bundlesdf: Neural 6-dof tracking and 3d reconstruction of unknown objects,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2023, pp. 606–617

2023

Cited alongside, same era.

C. Huang, O. Mees, A. Zeng, and W. Burgard, “Visual language maps for robot navigation,” in 2023 IEEE International Conference on Robotics and Automation (ICRA) , 2023, pp. 10 608–10 615

2023

Cited alongside, same era.

W. Cai, S. Huang, G. Cheng, Y. Long, P. Gao, C. Sun, and H. Dong, “Bridging zero-shot object navigation and foundation models through pixel-guided navigation skill,” 2023

2023

Cited alongside, same era.

J. Yang, H. Zhang, F. Li, X. Zou, C. Li, and J. Gao, “Set-of-mark prompting unleashes extraordinary visual grounding in gpt-4v,” 2023. [Online]. Available: https://arxiv.org/abs/2310.11441

2023

Cited alongside, same era.

L. Yang, Y. Wang, X. Li, X. Wang, and J. Yang, “Fine-grained visual prompting,” in Advances in Neural Information Processing Systems , A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, Eds., vol. 36. Curran Associates, Inc., 2023, pp. 24 993–25 006. [Online]. Available: https://proceedings.neurips.cc/paper_files/paper/2023/file/4e9fa6e716940a7cfc60c46e6f702f52-Paper-Conference.pdf

2023

Cited alongside, same era.

B. Chen, F. Xia, B. Ichter, K. Rao, K. Gopalakrishnan, M. S. Ryoo, A. Stone, and D. Kappler, “Open-vocabulary queryable scene representations for real world planning,” in 2023 IEEE International Conference on Robotics and Automation (ICRA) , 2023, pp. 11 509–11 522

2023

Cited alongside, same era.

A. Shtedritski, C. Rupprecht, and A. Vedaldi, “What does clip know about a red circle? visual prompt engineering for vlms,” in Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) , October 2023, pp. 11 987–11 997

2023

Cited alongside, same era.

D. Driess, F. Xia, M. S. M. Sajjadi, C. Lynch, A. Chowdhery, B. Ichter, A. Wahid, J. Tompson, Q. Vuong, T. Yu, W. Huang, Y. Chebotar, P. Sermanet, D. Duckworth, S. Levine, V. Vanhoucke, K. Hausman, M. Toussaint, K. Greff, A. Zeng, I. Mordatch, and P. Florence, “Palm-e: an embodied multimodal language model,” in Proceedings of the 40th International Conference on Machine Learning , ser. ICML’23. JMLR.org, 2023

2023

Later among the works it cites.

B. Yu, H. Kasaei, and M. Cao, “L3mvn: Leveraging large language models for visual target navigation,” in 2023 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) . IEEE, Oct. 2023. [Online]. Available: http://dx.doi.org/10.1109/IROS55552.2023.10342512

2023

Later among the works it cites.

K. Zhou, K. Zheng, C. Pryor, Y. Shen, H. Jin, L. Getoor, and X. E. Wang, “Esc: exploration with soft commonsense constraints for zero-shot object navigation,” in Proceedings of the 40th International Conference on Machine Learning , ser. ICML’23. JMLR.org, 2023

2023

Later among the works it cites.

D. Shah, M. Equi, B. Osinski, F. Xia, B. Ichter, and S. Levine, “Navigation with large language models: Semantic guesswork as a heuristic for planning,” in 7th Annual Conference on Robot Learning , 2023. [Online]. Available: https://openreview.net/forum?id=PsV65r0itpo

2023

Later among the works it cites.

J. Chen, G. Li, S. Kumar, B. Ghanem, and F. Yu, “How to not train your dragon: Training-free embodied object goal navigation with semantic frontiers,” 2023

2023

Later among the works it cites.

D. Shah, B. Osiński, S. Levine, et al. , “Lm-nav: Robotic navigation with large pre-trained models of language, vision, and action,” in Conference on robot learning . PMLR, 2023, pp. 492–504

2023

Later among the works it cites.

J. Zhu, H. Li, S. Wang, Z. Wang, and T. Zhang, “i-octree: A fast, lightweight, and dynamic octree for proximity search,” 2023

2023

Later among the works it cites.

OpenAI, “Gpt-4 technical report,” 2024

2024

Closest in time.

Y. Yao, A. Zhang, Z. Zhang, Z. Liu, T.-S. Chua, and M. Sun, “Cpt: Colorful prompt tuning for pre-trained vision-language models,” AI Open , vol. 5, pp. 30–38, 2024

2024

Closest in time.

V. S. Dorbala, J. F. Mullen, and D. Manocha, “Can an embodied agent find your “cat-shaped mug”? llm-based zero-shot object navigation,” IEEE Robotics and Automation Letters , vol. 9, no. 5, pp. 4083–4090, 2024

2024

Closest in time.