Fetching the paper…

Reading the bibliography…

2025

NavigateDiff: Visual Predictors are Zero-Shot Navigation Assistants

Qin, Yiran, Sun, Ao, Hong, Yuze et al.

Original
PDF
Post on X

Understand

Navigating unfamiliar environments presents significant challenges for household robots, requiring the ability to recognize and reason about novel decoration and layout.

Existing reinforcement learning methods cannot be directly transferred to new environments, as they typically rely on extensive mapping and exploration, leading to time-consuming and inefficient.
To address these challenges, we try to transfer the logical knowledge and the generalization ability of pre-trained foundation models to zero-shot navigation.
By integrating a large vision-language model with a diffusion network, our approach named \mname ~constructs a visual predictor that continuously predicts the agent's potential observations in the next step which can assist robots generate robust actions.

NavigateDiff: Visual Predictors are Zero-Shot Navigation Assistants · Around

Built on

Heusel, Martin, et al. ”Gans trained by a two time-scale update rule converge to a local nash equilibrium.” Advances in neural information processing systems 30 (2017)
2017
Earlier work this paper cites.
Zhu, Yuke, et al. ”Target-driven visual navigation in indoor scenes using deep reinforcement learning.” 2017 IEEE international conference on robotics and automation (ICRA). IEEE, 2017
2017
Earlier work this paper cites.
Schulman, John, et al. ”Proximal policy optimization algorithms.” arXiv preprint arXiv:1707.06347 (2017)
Original
2017
Earlier work this paper cites.
Zhang, Richard, et al. ”The unreasonable effectiveness of deep features as a perceptual metric.” Proceedings of the IEEE conference on computer vision and pattern recognition. 2018
2018
Earlier work this paper cites.
Savinov, Nikolay, Alexey Dosovitskiy, and Vladlen Koltun. ”Semi-parametric topological memory for navigation.” arXiv preprint arXiv:1803.00653 (2018)
Original
2018
Earlier work this paper cites.
Anderson, Peter, et al. ”On evaluation of embodied navigation agents.” arXiv preprint arXiv:1807.06757 (2018)
Original
2018
Earlier work this paper cites.
Mousavian, Arsalan, et al. ”Visual representations for semantic target driven navigation.” 2019 International Conference on Robotics and Automation (ICRA). IEEE, 2019
2019
Earlier work this paper cites.
Chaplot, Devendra Singh, et al. ”Neural topological slam for visual navigation.” Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2020
2020
Earlier work this paper cites.
Chaplot, Devendra Singh, et al. ”Object goal navigation using goal-oriented semantic exploration.” Advances in Neural Information Processing Systems 33 (2020)
2020
Earlier work this paper cites.
Brown, Tom B. ”Language models are few-shot learners.” arXiv preprint arXiv:2005.14165 (2020)
Original
2020
Earlier work this paper cites.
Maksymets, Oleksandr, et al. ”Thda: Treasure hunt data augmentation for semantic navigation.” Proceedings of the IEEE/CVF International Conference on Computer Vision. 2021
2021
Earlier work this paper cites.
Ye, Joel, et al. ”Auxiliary tasks and exploration enable objectgoal navigation.” Proceedings of the IEEE/CVF international conference on computer vision. 2021
2021
Earlier work this paper cites.
Dhariwal, Prafulla, and Alexander Nichol. ”Diffusion models beat gans on image synthesis.” Advances in neural information processing systems 34 (2021)
2021
Earlier work this paper cites.
Nichol, Alex, et al. ”Glide: Towards photorealistic image generation and editing with text-guided diffusion models.” arXiv preprint arXiv:2112.10741 (2021)
Original
2021
Earlier work this paper cites.
Hu, Edward J., et al. ”Lora: Low-rank adaptation of large language models.” arXiv preprint arXiv:2106.09685 (2021)
Original
2021
Earlier work this paper cites.
Hahn, Meera, et al. ”No rl, no simulation: Learning to navigate without navigating.” Advances in Neural Information Processing Systems 34 (2021): 26661-26673
2021
Earlier work this paper cites.
Mezghan, Lina, et al. ”Memory-augmented reinforcement learning for image-goal navigation.” 2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). IEEE, 2022
2022
Earlier work this paper cites.
Al-Halah, Ziad, Santhosh Kumar Ramakrishnan, and Kristen Grauman. ”Zero experience required: Plug & play modular transfer learning for semantic visual navigation.” Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022
2022
Earlier work this paper cites.
Ramakrishnan, Santhosh Kumar, et al. ”Poni: Potential functions for objectgoal navigation with interaction-free learning.” Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022
2022
Earlier work this paper cites.
Ramrakhya, Ram, et al. ”Habitat-web: Learning embodied object-search strategies from human demonstrations at scale.” Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022
2022
Earlier work this paper cites.

Similar

Ho, Jonathan, and Tim Salimans. ”Classifier-free diffusion guidance.” arXiv preprint arXiv:2207.12598 (2022)
Original
2022
Cited alongside, same era.
Brohan, Anthony, et al. ”Rt-1: Robotics transformer for real-world control at scale.” arXiv preprint arXiv:2212.06817 (2022)
Original
2022
Cited alongside, same era.
Majumdar, Arjun, et al. ”Zson: Zero-shot object-goal navigation using multimodal goal embeddings.” Advances in Neural Information Processing Systems 35 (2022): 32340-32352
2022
Cited alongside, same era.
Krantz, Jacob, et al. ”Instance-specific image goal navigation: Training embodied agents to find object instances.” arXiv preprint arXiv:2211.15876 (2022)
Original
2022
Cited alongside, same era.

Then

Zhou, Enshen, et al. ”Minedreamer: Learning to follow instructions via chain-of-imagination for simulated-world control.” arXiv preprint arXiv:2403.12037 (2024)
Original
2024
Later among the works it cites.
Cai, Wenzhe, et al. ”Bridging zero-shot object navigation and foundation models through pixel-guided navigation skill.” 2024 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2024
2024
Later among the works it cites.
Wu, Pengying, et al. ”Voronav: Voronoi-based zero-shot object navigation with large language model.” arXiv preprint arXiv:2401.02695 (2024)
Original
2024
Later among the works it cites.
Du, Yilun, et al. ”Learning universal policies via text-guided video generation.” Advances in Neural Information Processing Systems 36 (2024)
2024
Later among the works it cites.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Brooks, Tim, Aleksander Holynski, and Alexei A. Efros. ”Instructpix2pix: Learning to follow image editing instructions.” Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023

2023

Cited alongside, same era.

Fu, Tsu-Jui, et al. ”Guiding instruction-based image editing via multimodal large language models.” arXiv preprint arXiv:2309.17102 (2023)

2023

Cited alongside, same era.

Zhou, Kaiwen, et al. ”Esc: Exploration with soft commonsense constraints for zero-shot object navigation.” International Conference on Machine Learning. PMLR, 2023

2023

Cited alongside, same era.

Yu, Bangguo, Hamidreza Kasaei, and Ming Cao. ”L3mvn: Leveraging large language models for visual target navigation.” 2023 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). IEEE, 2023

2023

Cited alongside, same era.

Gadre, Samir Yitzhak, et al. ”Cows on pasture: Baselines and benchmarks for language-driven zero-shot object navigation.” Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023

2023

Cited alongside, same era.

Shah, Dhruv, et al. ”Navigation with large language models: Semantic guesswork as a heuristic for planning.” Conference on Robot Learning. PMLR, 2023

2023

Cited alongside, same era.

Yu, Bangguo, Hamidreza Kasaei, and Ming Cao. ”Co-NavGPT: Multi-robot cooperative visual semantic navigation using large language models.” arXiv preprint arXiv:2310.07937 (2023)

2023

Cited alongside, same era.

Qin, Yiran, et al. ”Mp5: A multi-modal open-ended embodied system in minecraft via active perception.” arXiv preprint arXiv:2312.07472 (2023)

2023

Cited alongside, same era.

Ajay, Anurag, et al. ”Compositional foundation models for hierarchical planning.” Advances in Neural Information Processing Systems 36 (2024)

2024

Later among the works it cites.

Liang, Zhixuan, et al. ”Skilldiffuser: Interpretable hierarchical planning via skill abstractions in diffusion-based task execution.” Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024

2024

Later among the works it cites.

Li, Xiaoqi, et al. ”Manipllm: Embodied multimodal large language model for object-centric robotic manipulation.” Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024

2024

Later among the works it cites.

Liu, Haotian, et al. ”Visual instruction tuning.” Advances in neural information processing systems 36 (2024)

2024

Later among the works it cites.

Qin, Yiran, et al. ”Worldsimbench: Towards video generation models as world simulators.” arXiv preprint arXiv:2410.18072 (2024)

2024

Later among the works it cites.

Zhou, Enshen, et al. ”Code-as-Monitor: Constraint-aware Visual Programming for Reactive and Proactive Robotic Failure Detection.” arXiv preprint arXiv:2412.04455 (2024)

2024

Later among the works it cites.

Zhang, Zaibin, et al. ”AD-H: Autonomous Driving with Hierarchical Agents.” arXiv preprint arXiv:2406.03474 (2024)

2024

Later among the works it cites.

Wang, Chaoqun, et al. ”Toward Accurate Camera-based 3D Object Detection via Cascade Depth Estimation and Calibration.” arXiv preprint arXiv:2402.04883 (2024)

2024

Later among the works it cites.

Huang, Yuzhou, et al. ”Story3d-agent: Exploring 3d storytelling visualization with large language models.” arXiv preprint arXiv:2408.11801 (2024)

2024

Later among the works it cites.

Sun, Xinyu, et al. ”FGPrompt: fine-grained goal prompting for image-goal navigation.” Advances in Neural Information Processing Systems 36 (2024)

2024

Later among the works it cites.

Koh, Jing Yu, Daniel Fried, and Russ R. Salakhutdinov. ”Generating images with multimodal language models.” Advances in Neural Information Processing Systems 36 (2024)

2024

Later among the works it cites.

Lin, Bingqian, et al. ”NavCoT: Boosting LLM-Based Vision-and-Language Navigation via Learning Disentangled Reasoning.” arXiv preprint arXiv:2403.07376 (2024)

2024

Later among the works it cites.

An, Jingkun, et al. ”AGFSync: Leveraging AI-Generated Feedback for Preference Optimization in Text-to-Image Generation.” arXiv preprint arXiv:2403.13352 (2024)

2024

Later among the works it cites.

Yu, Jiwen, et al. ”GameFactory: Creating New Games with Generative Interactive Videos.” arXiv preprint arXiv:2501.08325 (2025)

2025

Closest in time.

Li, Lijun, et al. ”T2ISafety: Benchmark for Assessing Fairness, Toxicity, and Privacy in Image Generation.” arXiv preprint arXiv:2501.12612 (2025)

2025

Closest in time.