Fetching the paper…

Reading the bibliography…

2025

Learning Affordances at Inference-Time for Vision-Language-Action Models

Shah, Ameesh, Chen, William, Godbole, Adwait et al.

Original
PDF
Post on X

Understand

Solving complex real-world control tasks often takes multiple tries: if we fail at first, we reflect on what went wrong, and change our strategy accordingly to avoid making the same mistake.

In robotics, Vision-Language-Action models (VLAs) offer a promising path towards solving complex control tasks, but lack the ability to contextually and dynamically readjust behavior when they fail to accomplish a task.
In this work, we introduce Learning from Inference-Time Execution (LITEN), which connects a VLA low-level policy to a high-level VLM that conditions on past experiences by including them in-context, allowing it to learn the affordances and capabilities of the low-level VLA.
Our approach iterates between a reasoning phase that generates and executes plans for the low-level VLA, and an assessment phase that reflects on the resulting execution and draws useful conclusions to be included in future reasoning contexts.

Learning Affordances at Inference-Time for Vision-Language-Action Models · Around

Built on

J. J. Gibson, The Ecological Approach to Visual Perception: Classic Edition . Psychology Press, 2014
2014
Earlier work this paper cites.
P. Lewis, E. Perez, A. Piktus, F. Petroni, V. Karpukhin, N. Goyal, H. Küttler, M. Lewis, W.-t. Yih, T. Rocktäschel, et al. , “Retrieval-augmented generation for knowledge-intensive NLP tasks,” Advances in neural information processing systems , vol. 33, pp. 9459–9474, 2020
2020
Earlier work this paper cites.
M. Ahn, A. Brohan, N. Brown, Y. Chebotar, O. Cortes, B. David, C. Finn, C. Fu, K. Gopalakrishnan, K. Hausman, et al. , “Do as i can, not as i say: Grounding language in robotic affordances,” arXiv preprint arXiv:2204.01691 , 2022
Original
2022
Earlier work this paper cites.
W. Huang, F. Xia, T. Xiao, H. Chan, J. Liang, P. Florence, A. Zeng, J. Tompson, I. Mordatch, Y. Chebotar, et al. , “Inner monologue: Embodied reasoning through planning with language models,” arXiv preprint arXiv:2207.05608 , 2022
Original
2022
Earlier work this paper cites.
A. Zeng, M. Attarian, B. Ichter, K. Choromanski, A. Wong, S. Welker, F. Tombari, A. Purohit, M. Ryoo, V. Sindhwani, J. Lee, V. Vanhoucke, and P. Florence, “Socratic models: Composing zero-shot multimodal reasoning with language,” 2022
2022
Earlier work this paper cites.
B. Ichter, A. Brohan, et al. , “Do as I can, not as I say: Grounding language in robotic affordances,” in Conference on Robot Learning, CoRL 2022, 14-18 December 2022, Auckland, New Zealand , ser. Proceedings of Machine Learning Research, K. Liu, D. Kulic, and J. Ichnowski, Eds., vol. 205. PMLR, 2022, pp. 287–318. [Online]. Available: https://proceedings.mlr.press/v205/ichter23a.html
2022
Earlier work this paper cites.
B. Chen, F. Xia, B. Ichter, K. Rao, K. Gopalakrishnan, M. S. Ryoo, A. Stone, and D. Kappler, “Open-vocabulary queryable scene representations for real world planning,” arXiv preprint arXiv:2209.09874 , 2022
Original
2022
Earlier work this paper cites.
A. Brohan, N. Brown, J. Carbajal, Y. Chebotar, X. Chen, K. Choromanski, T. Ding, D. Driess, A. Dubey, C. Finn, et al. , “Rt-2: Vision-language-action models transfer web knowledge to robotic control,” arXiv preprint arXiv:2307.15818 , 2023
Original
2023
Earlier work this paper cites.
Octo Model Team, D. Ghosh, H. Walke, K. Pertsch, K. Black, O. Mees, S. Dasari, J. Hejna, C. Xu, J. Luo, T. Kreiman, Y. Tan, D. Sadigh, C. Finn, and S. Levine, “Octo: An open-source generalist robot policy,” https://octo-models.github.io , 2023
2023
Earlier work this paper cites.
A. Madaan, N. Tandon, P. Gupta, S. Hallinan, L. Gao, S. Wiegreffe, U. Alon, N. Dziri, S. Prabhumoye, Y. Yang, et al. , “Self-refine: Iterative refinement with self-feedback,” Advances in Neural Information Processing Systems , vol. 36, pp. 46 534–46 594, 2023
2023
Earlier work this paper cites.
Z. Gou, Z. Shao, Y. Gong, Y. Shen, Y. Yang, N. Duan, and W. Chen, “Critic: Large language models can self-correct with tool-interactive critiquing,” arXiv preprint arXiv:2305.11738 , 2023
Original
2023
Earlier work this paper cites.
N. Shinn, F. Cassano, A. Gopinath, K. Narasimhan, and S. Yao, “Reflexion: Language agents with verbal reinforcement learning,” Advances in Neural Information Processing Systems , vol. 36, pp. 8634–8652, 2023
2023
Earlier work this paper cites.
S. An, Z. Ma, Z. Lin, N. Zheng, J.-G. Lou, and W. Chen, “Learning from mistakes makes LLM better reasoner,” arXiv preprint arXiv:2310.20689 , 2023
Original
2023
Earlier work this paper cites.
H. R. Walke, K. Black, T. Z. Zhao, Q. Vuong, C. Zheng, P. Hansen-Estruch, A. W. He, V. Myers, M. J. Kim, M. Du, A. Lee, K. Fang, C. Finn, and S. Levine, “Bridgedata V2: A dataset for robot learning at scale,” in Conference on Robot Learning, CoRL 2023, 6-9 November 2023, Atlanta, GA, USA , ser. Proceedings of Machine Learning Research, J. Tan, M. Toussaint, and K. Darvish, Eds., vol. 229. PMLR, 2023, pp. 1723–1736. [Online]. Available: https://proceedings.mlr.press/v229/walke23a.html
2023
Earlier work this paper cites.

Similar

W. Huang, C. Wang, R. Zhang, Y. Li, J. Wu, and L. Fei-Fei, “Voxposer: Composable 3D Value Maps for Robotic Manipulation with Language Models,” arXiv preprint arXiv:2307.05973 , 2023
Original
2023
Cited alongside, same era.
J. Liang, W. Huang, F. Xia, P. Xu, K. Hausman, B. Ichter, P. Florence, and A. Zeng, “Code as Policies: Language Model Programs for Embodied Control,” in IEEE International Conference on Robotics and Automation . IEEE, 2023, pp. 9493–9500
2023
Cited alongside, same era.
Y. Du, O. Watkins, Z. Wang, C. Colas, T. Darrell, P. Abbeel, A. Gupta, and J. Andreas, “Guiding pretraining in reinforcement learning with large language models,” 2023
2023
Cited alongside, same era.
G. Wang, Y. Xie, Y. Jiang, A. Mandlekar, C. Xiao, Y. Zhu, L. Fan, and A. Anandkumar, “Voyager: An open-ended embodied agent with large language models,” arXiv preprint arXiv:2305.16291 , 2023
Original
2023
Cited alongside, same era.

Then

V. Vosylius and E. Johns, “Instant policy: In-context imitation learning via graph diffusion,” arXiv preprint arXiv:2411.12633 , 2024
Original
2024
Later among the works it cites.
L. Fu, H. Huang, G. Datta, L. Y. Chen, W. C.-H. Panitch, F. Liu, H. Li, and K. Goldberg, “In-context imitation learning via next-token prediction,” arXiv preprint arXiv:2408.15980 , 2024
Original
2024
Later among the works it cites.
Z. Jiang, Y. Xie, K. Lin, Z. Xu, W. Wan, A. Mandlekar, L. J. Fan, and Y. Zhu, “Dexmimicgen: Automated data generation for bimanual dexterous manipulation via imitation learning,” in 2025 IEEE International Conference on Robotics and Automation (ICRA) . IEEE, 2025, pp. 16 923–16 930
2025
Closest in time.
J. Bjorck, F. Castañeda, N. Cherniadev, X. Da, R. Ding, L. Fan, Y. Fang, D. Fox, F. Hu, S. Huang, et al. , “Gr00t n1: An open foundation model for generalist humanoid robots,” arXiv preprint arXiv:2503.14734 , 2025
Original
2025
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

T. Z. Zhao, V. Kumar, S. Levine, and C. Finn, “Learning fine-grained bimanual manipulation with low-cost hardware,” in Robotics: Science and Systems XIX, Daegu, Republic of Korea, July 10-14, 2023 , K. E. Bekris, K. Hauser, S. L. Herbert, and J. Yu, Eds., 2023. [Online]. Available: https://doi.org/10.15607/RSS.2023.XIX.016

2023

Cited alongside, same era.

K. Black, N. Brown, D. Driess, A. Esmail, M. Equi, C. Finn, N. Fusai, L. Groom, K. Hausman, B. Ichter, et al. , “ π 0 \pi_{0} : A vision-language-action flow model for general robot control,” arXiv preprint arXiv:2410.24164 , 2024

2024

Cited alongside, same era.

M. J. Kim, K. Pertsch, S. Karamcheti, T. Xiao, A. Balakrishna, S. Nair, R. Rafailov, E. P. Foster, P. R. Sanketi, Q. Vuong, T. Kollar, B. Burchfiel, R. Tedrake, D. Sadigh, S. Levine, P. Liang, and C. Finn, “Openvla: An open-source vision-language-action model,” in Conference on Robot Learning, 6-9 November 2024, Munich, Germany , ser. Proceedings of Machine Learning Research, P. Agrawal, O. Kroemer, and W. Burgard, Eds., vol. 270. PMLR, 2024, pp. 2679–2713. [Online]. Available: https://proceedings.mlr.press/v270/kim25c.html

2024

Cited alongside, same era.

Y. Tong, D. Li, S. Wang, Y. Wang, F. Teng, and J. Shang, “Can LLMs learn from previous mistakes? investigating LLMs’ errors to boost for reasoning,” arXiv preprint arXiv:2403.20046 , 2024

2024

Cited alongside, same era.

A. Khazatsky, K. Pertsch, et al. , “DROID: A large-scale in-the-wild robot manipulation dataset,” in Robotics: Science and Systems XX, Delft, The Netherlands, July 15-19, 2024 , D. Kulic, G. Venture, K. E. Bekris, and E. Coronado, Eds., 2024. [Online]. Available: https://doi.org/10.15607/RSS.2024.XX.120

2024

Cited alongside, same era.

A. O’Neill, A. Rehman, A. Maddukuri, A. Gupta, A. Padalkar, A. Lee, A. Pooley, A. Gupta, A. Mandlekar, A. Jain, et al. , “Open x-embodiment: Robotic learning datasets and rt-x models: Open x-embodiment collaboration 0,” in 2024 IEEE International Conference on Robotics and Automation (ICRA) . IEEE, 2024, pp. 6892–6903

2024

Cited alongside, same era.

M. Zawalski, W. Chen, K. Pertsch, O. Mees, C. Finn, and S. Levine, “Robotic control via embodied chain-of-thought reasoning,” 2024

2024

Cited alongside, same era.

M. Nakamoto, O. Mees, A. Kumar, and S. Levine, “Steering your generalists: Improving robotic foundation models via value guidance,” arXiv preprint arXiv:2410.13816 , 2024

2024

Cited alongside, same era.

P. Intelligence et al. , “ π 0.5 \pi_{0.5} : a vision-language-action model with open-world generalization,” 2025

2025

Closest in time.

L. X. Shi, B. Ichter, M. Equi, L. Ke, K. Pertsch, Q. Vuong, J. Tanner, A. Walling, H. Wang, N. Fusai, et al. , “Hi robot: Open-ended instruction following with hierarchical vision-language-action models,” arXiv preprint arXiv:2502.19417 , 2025

2025

Closest in time.

Gemini Robotics Team and others, “Gemini robotics: Bringing AI into the physical world,” 2025

2025

Closest in time.

W. Chen, S. Belkhale, S. Mirchandani, O. Mees, D. Driess, K. Pertsch, and S. Levine, “Training strategies for efficient embodied reasoning,” 2025

2025

Closest in time.

F. Lin, R. Nai, Y. Hu, J. You, J. Zhao, and Y. Gao, “Onetwovla: A unified vision-language-action model with adaptive reasoning,” CoRR , vol. abs/2505.11917, 2025. [Online]. Available: https://doi.org/10.48550/arXiv.2505.11917

2025

Closest in time.

Y. J. Ma, J. Hejna, C. Fu, D. Shah, J. Liang, Z. Xu, S. Kirmani, P. Xu, D. Driess, T. Xiao, O. Bastani, D. Jayaraman, W. Yu, T. Zhang, D. Sadigh, and F. Xia, “Vision language models are in-context value learners,” in The Thirteenth International Conference on Learning Representations, ICLR 2025, Singapore, April 24-28, 2025 . OpenReview.net, 2025. [Online]. Available: https://openreview.net/forum?id=friHAl5ofG

2025

Closest in time.

L. Smith, A. Irpan, M. G. Arenas, S. Kirmani, D. Kalashnikov, D. Shah, and T. Xiao, “Steer: Flexible robotic manipulation via dense language grounding,” in 2025 IEEE International Conference on Robotics and Automation (ICRA) . IEEE, 2025, pp. 16 517–16 524

2025

Closest in time.

OpenAI et al. , “GPT-5 System Card,” OpenAI Blog , 2025

2025

Closest in time.

C. Glossop, W. Chen, A. Bhorkar, D. Shah, and S. Levine, “Cast: Counterfactual labels improve instruction following in vision-language-action models,” arXiv preprint arXiv:2508.13446 , 2025

2025

Closest in time.