Built on

R. S. Sutton, D. Precup, and S. Singh, “Between mdps and semi-mdps: A framework for temporal abstraction in reinforcement learning,” Artificial intelligence , vol. 112, no. 1-2, pp. 181–211, 1999
1999
Earlier work this paper cites.
B. Calli, A. Singh, A. Walsman, S. Srinivasa, P. Abbeel, and A. M. Dollar, “The ycb object and model set: Towards common benchmarks for manipulation research,” in 2015 international conference on advanced robotics (ICAR) . IEEE, 2015, pp. 510–517
2015
Earlier work this paper cites.
J. Bohg, K. Hausman, B. Sankaran, O. Brock, D. Kragic, S. Schaal, and G. S. Sukhatme, “Interactive perception: Leveraging action in perception and perception in action,” IEEE Transactions on Robotics , vol. 33, no. 6, pp. 1273–1291, 2017
2017
Earlier work this paper cites.
P. Anderson, Q. Wu, D. Teney, J. Bruce, M. Johnson, N. Sünderhauf, I. Reid, S. Gould, and A. Van Den Hengel, “Vision-and-language navigation: Interpreting visually-grounded navigation instructions in real environments,” in Proceedings of the IEEE conference on computer vision and pattern recognition , 2018, pp. 3674–3683
Original
2018
Earlier work this paper cites.
Y. Bisk, R. Zellers, J. Gao, Y. Choi et al. , “Piqa: Reasoning about physical commonsense in natural language,” in Proceedings of the AAAI conference on artificial intelligence , vol. 34, no. 05, 2020, pp. 7432–7439
2020
Earlier work this paper cites.
S. Aroca-Ouellette, C. Paik, A. Roncone, and K. Kann, “Prost: Physical reasoning of objects through space and time,” arXiv preprint arXiv:2106.03634 , 2021
Original
2021
Earlier work this paper cites.
O. Kroemer, S. Niekum, and G. Konidaris, “A review of robot learning for manipulation: Challenges, representations, and algorithms,” Journal of machine learning research , vol. 22, no. 30, pp. 1–82, 2021
2021
Earlier work this paper cites.

Similar

P. Sharma, A. Torralba, and J. Andreas, “Skill induction and planning with latent language,” arXiv preprint arXiv:2110.01517 , 2021
Original
2021
Cited alongside, same era.
M. Ahn, A. Brohan, N. Brown, Y. Chebotar, O. Cortes, B. David, C. Finn, C. Fu, K. Gopalakrishnan, K. Hausman, A. Herzog, D. Ho, J. Hsu, J. Ibarz, B. Ichter, A. Irpan, E. Jang, R. J. Ruano, K. Jeffrey, S. Jesmonth, N. Joshi, R. Julian, D. Kalashnikov, Y. Kuang, K.-H. Lee, S. Levine, Y. Lu, L. Luu, C. Parada, P. Pastor, J. Quiambao, K. Rao, J. Rettinghouse, D. Reyes, P. Sermanet, N. Sievers, C. Tan, A. Toshev, V. Vanhoucke, F. Xia, T. Xiao, P. Xu, S. Xu, M. Yan, and A. Zeng, “Do as i can and not as i say: Grounding language in robotic affordances,” in arXiv preprint arXiv:2204.01691 , 2022
Original
2022
Cited alongside, same era.
H. Wang, J. Zhang, Y. Chen, C. Ma, J. Avery, L. Hull, and G. Carneiro, “Uncertainty-aware multi-modal learning via cross-modal random network prediction,” in European Conference on Computer Vision . Springer, 2022, pp. 200–217
2022
Cited alongside, same era.
J. Liang, W. Huang, F. Xia, P. Xu, K. Hausman, B. Ichter, P. Florence, and A. Zeng, “Code as policies: Language model programs for embodied control,” in 2023 IEEE International Conference on Robotics and Automation (ICRA) . IEEE, 2023, pp. 9493–9500

Then

C. Celemin and J. Kober, “Knowledge-and ambiguity-aware robot learning from corrective and evaluative feedback,” Neural Computing and Applications , vol. 35, no. 23, pp. 16 821–16 839, 2023
2023
Later among the works it cites.
M. Li, X. Zhao, J. H. Lee, C. Weber, and S. Wermter, “Internally rewarded reinforcement learning,” in International Conference on Machine Learning . PMLR, 2023, pp. 20 556–20 574
2023
Later among the works it cites.
B. Chen, Z. Xu, S. Kirmani, B. Ichter, D. Driess, P. Florence, D. Sadigh, L. Guibas, and F. Xia, “Spatialvlm: Endowing vision-language models with spatial reasoning capabilities,” arXiv preprint arXiv:2401.12168 , 2024. [Online]. Available: https://arxiv.org/abs/2401.12168
Original
2024
Closest in time.
F. Liu, K. Fang, P. Abbeel, and S. Levine, “Moka: Open-world robotic manipulation through mark-based visual prompting,” 2024. [Online]. Available: https://arxiv.org/abs/2403.03174
Original
2024
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…

2024

Scene Exploration by Vision-Language Models

Sripada, Venkatesh, Carter, Samuel, Guerin, Frank et al.

Understand

Active perception enables robots to dynamically gather information by adjusting their viewpoints, a crucial capability for interacting with complex, partially observable environments.

In this paper, we present AP-VLM, a novel framework that combines active perception with a Vision-Language Model (VLM) to guide robotic exploration and answer semantic queries.
Using a 3D virtual grid overlaid on the scene and orientation adjustments, AP-VLM allows a robotic manipulator to intelligently select optimal viewpoints and orientations to resolve challenging tasks, such as identifying objects in occluded or inclined positions.
We evaluate our system on two robotic platforms: a 7-DOF Franka Panda and a 6-DOF UR5, across various scenes with differing object configurations.

Built on

R. S. Sutton, D. Precup, and S. Singh, “Between mdps and semi-mdps: A framework for temporal abstraction in reinforcement learning,” Artificial intelligence , vol. 112, no. 1-2, pp. 181–211, 1999
1999
Earlier work this paper cites.
B. Calli, A. Singh, A. Walsman, S. Srinivasa, P. Abbeel, and A. M. Dollar, “The ycb object and model set: Towards common benchmarks for manipulation research,” in 2015 international conference on advanced robotics (ICAR) . IEEE, 2015, pp. 510–517
2015
Earlier work this paper cites.
J. Bohg, K. Hausman, B. Sankaran, O. Brock, D. Kragic, S. Schaal, and G. S. Sukhatme, “Interactive perception: Leveraging action in perception and perception in action,” IEEE Transactions on Robotics , vol. 33, no. 6, pp. 1273–1291, 2017
2017
Earlier work this paper cites.
P. Anderson, Q. Wu, D. Teney, J. Bruce, M. Johnson, N. Sünderhauf, I. Reid, S. Gould, and A. Van Den Hengel, “Vision-and-language navigation: Interpreting visually-grounded navigation instructions in real environments,” in Proceedings of the IEEE conference on computer vision and pattern recognition , 2018, pp. 3674–3683
Original
2018
Earlier work this paper cites.
Y. Bisk, R. Zellers, J. Gao, Y. Choi et al. , “Piqa: Reasoning about physical commonsense in natural language,” in Proceedings of the AAAI conference on artificial intelligence , vol. 34, no. 05, 2020, pp. 7432–7439
2020
Earlier work this paper cites.
S. Aroca-Ouellette, C. Paik, A. Roncone, and K. Kann, “Prost: Physical reasoning of objects through space and time,” arXiv preprint arXiv:2106.03634 , 2021
Original
2021
Earlier work this paper cites.
O. Kroemer, S. Niekum, and G. Konidaris, “A review of robot learning for manipulation: Challenges, representations, and algorithms,” Journal of machine learning research , vol. 22, no. 30, pp. 1–82, 2021
2021
Earlier work this paper cites.

Similar

P. Sharma, A. Torralba, and J. Andreas, “Skill induction and planning with latent language,” arXiv preprint arXiv:2110.01517 , 2021
Original
2021
Cited alongside, same era.
M. Ahn, A. Brohan, N. Brown, Y. Chebotar, O. Cortes, B. David, C. Finn, C. Fu, K. Gopalakrishnan, K. Hausman, A. Herzog, D. Ho, J. Hsu, J. Ibarz, B. Ichter, A. Irpan, E. Jang, R. J. Ruano, K. Jeffrey, S. Jesmonth, N. Joshi, R. Julian, D. Kalashnikov, Y. Kuang, K.-H. Lee, S. Levine, Y. Lu, L. Luu, C. Parada, P. Pastor, J. Quiambao, K. Rao, J. Rettinghouse, D. Reyes, P. Sermanet, N. Sievers, C. Tan, A. Toshev, V. Vanhoucke, F. Xia, T. Xiao, P. Xu, S. Xu, M. Yan, and A. Zeng, “Do as i can and not as i say: Grounding language in robotic affordances,” in arXiv preprint arXiv:2204.01691 , 2022
Original
2022
Cited alongside, same era.
H. Wang, J. Zhang, Y. Chen, C. Ma, J. Avery, L. Hull, and G. Carneiro, “Uncertainty-aware multi-modal learning via cross-modal random network prediction,” in European Conference on Computer Vision . Springer, 2022, pp. 200–217
2022
Cited alongside, same era.
J. Liang, W. Huang, F. Xia, P. Xu, K. Hausman, B. Ichter, P. Florence, and A. Zeng, “Code as policies: Language model programs for embodied control,” in 2023 IEEE International Conference on Robotics and Automation (ICRA) . IEEE, 2023, pp. 9493–9500

Then

C. Celemin and J. Kober, “Knowledge-and ambiguity-aware robot learning from corrective and evaluative feedback,” Neural Computing and Applications , vol. 35, no. 23, pp. 16 821–16 839, 2023
2023
Later among the works it cites.
M. Li, X. Zhao, J. H. Lee, C. Weber, and S. Wermter, “Internally rewarded reinforcement learning,” in International Conference on Machine Learning . PMLR, 2023, pp. 20 556–20 574
2023
Later among the works it cites.
B. Chen, Z. Xu, S. Kirmani, B. Ichter, D. Driess, P. Florence, D. Sadigh, L. Guibas, and F. Xia, “Spatialvlm: Endowing vision-language models with spatial reasoning capabilities,” arXiv preprint arXiv:2401.12168 , 2024. [Online]. Available: https://arxiv.org/abs/2401.12168
Original
2024
Closest in time.
F. Liu, K. Fang, P. Abbeel, and S. Levine, “Moka: Open-world robotic manipulation through mark-based visual prompting,” 2024. [Online]. Available: https://arxiv.org/abs/2403.03174
Original
2024
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

2023

Cited alongside, same era.

Y. R. Wang, J. Duan, D. Fox, and S. Srinivasa, “Newton: Are large language models capable of physical reasoning?” 2023

2023

Cited alongside, same era.

I. Singh, V. Blukis, A. Mousavian, A. Goyal, D. Xu, J. Tremblay, D. Fox, J. Thomason, and A. Garg, “Progprompt: Generating situated robot task plans using large language models,” in 2023 IEEE International Conference on Robotics and Automation (ICRA) . IEEE, 2023, pp. 11 523–11 530

2023

Cited alongside, same era.

J. Wu, R. Antonova, A. Kan, M. Lepert, A. Zeng, S. Song, J. Bohg, S. Rusinkiewicz, and T. Funkhouser, “Tidybot: Personalized robot assistance with large language models,” Autonomous Robots , vol. 47, no. 8, pp. 1087–1102, 2023

2023

Cited alongside, same era.

S. Nasiriany, F. Xia, W. Yu, T. Xiao, J. Liang, I. Dasgupta, A. Xie, D. Driess, A. Wahid, Z. Xu, Q. Vuong, T. Zhang, T.-W. E. Lee, K.-H. Lee, P. Xu, S. Kirmani, Y. Zhu, A. Zeng, K. Hausman, N. Heess, C. Finn, S. Levine, and B. Ichter, “Pivot: Iterative visual prompting elicits actionable knowledge for vlms,” 2024. [Online]. Available: https://arxiv.org/abs/2402.07872

Original

2024

Closest in time.

M. Kwon, H. Hu, V. Myers, S. Karamcheti, A. Dragan, and D. Sadigh, “Toward grounded commonsense reasoning,” 2024. [Online]. Available: https://arxiv.org/abs/2306.08651

Original

2024

Closest in time.