VLM See, Robot Do: Human Demo Video to Robot Action Plan via Vision Language Model

M. Kaiser and R. Dillmann, “Building elementary robot skills from human demonstration,” in

1996

Earlier work this paper cites.

R. Dillmann, “Teaching and learning of robot tasks via observation of human performance,”

2004

Earlier work this paper cites.

E. Coumans and Y. Bai, “Pybullet, a python module for physics simulation for games, robotics and machine learning,” 2016

2016

Earlier work this paper cites.

Y. Duan, M. Andrychowicz, B. Stadie, O. Jonathan Ho, J. Schneider, I. Sutskever, P. Abbeel, and W. Zaremba, “One-shot imitation learning,”

2017

Earlier work this paper cites.

T. Yu, C. Finn, A. Xie, S. Dasari, T. Zhang, P. Abbeel, and S. Levine, “One-shot imitation from observing humans via domain-adaptive meta-learning,”

Original

2018

Earlier work this paper cites.

S. Yang, W. Zhang, W. Lu, H. Wang, and Y. Li, “Learning actions from human demonstration video for robotic manipulation,”

Original

2019

Earlier work this paper cites.

C. Lugaresi, J. Tang, H. Nash, C. McClanahan, E. Uboweja, M. Hays, F. Zhang, C.-L. Chang, M. G. Yong, J. Lee,

Original

2019

Earlier work this paper cites.

T. B. Brown, “Language models are few-shot learners,”

Original

2020

Earlier work this paper cites.

S. Tellex, N. Gopalan, H. Kress-Gazit, and C. Matuszek, “Robots that use language,”

2020

Earlier work this paper cites.

A. Bonardi, S. James, and A. J. Davison, “Learning one-shot imitation from humans without humans,”

2020

Earlier work this paper cites.

I. Yanokura, N. Wake, K. Sasabuchi, K. Ikeuchi, and M. Inaba, “Understanding action sequences based on video captioning for learning-from-observation,”

Original

2020

Earlier work this paper cites.

M. Ahn, A. Brohan, N. Brown, Y. Chebotar, O. Cortes, B. David, C. Finn, C. Fu, K. Gopalakrishnan, K. Hausman, A. Herzog, D. Ho, J. Hsu, J. Ibarz, B. Ichter, A. Irpan, E. Jang, R. J. Ruano, K. Jeffrey, S. Jesmonth, N. Joshi, R. Julian, D. Kalashnikov, Y. Kuang, K.-H. Lee, S. Levine, Y. Lu, L. Luu, C. Parada, P. Pastor, J. Quiambao, K. Rao, J. Rettinghouse, D. Reyes, P. Sermanet, N. Sievers, C. Tan, A. Toshev, V. Vanhoucke, F. Xia, T. Xiao, P. Xu, S. Xu, M. Yan, and A. Zeng, “Do as i can and not as i say: Grounding language in robotic affordances,” in

Original

2022

Earlier work this paper cites.

W. Huang, P. Abbeel, D. Pathak, and I. Mordatch, “Language models as zero-shot planners: Extracting actionable knowledge for embodied agents,” in

2022

Earlier work this paper cites.

J. Wei, X. Wang, D. Schuurmans, M. Bosma, F. Xia, E. Chi, Q. V. Le, D. Zhou,

2022

Earlier work this paper cites.

W. Huang, F. Xia, T. Xiao, H. Chan, J. Liang, P. Florence, A. Zeng, J. Tompson, I. Mordatch, Y. Chebotar,

Original

2022

Earlier work this paper cites.

S. Nair, A. Rajeswaran, V. Kumar, C. Finn, and A. Gupta, “R3m: A universal visual representation for robot manipulation,”

Original

2022

Earlier work this paper cites.

S. Bahl, A. Gupta, and D. Pathak, “Human-to-robot imitation in the wild,”

2022

Earlier work this paper cites.

Z. Mandi, F. Liu, K. Lee, and P. Abbeel, “Towards more generalizable one-shot visual imitation learning,” in

2022

Earlier work this paper cites.

H. Bahng, A. Jahanian, S. Sankaranarayanan, and P. Isola, “Exploring visual prompts for adapting large-scale models,”

Original

2022

Earlier work this paper cites.

A. Chowdhery, S. Narang, J. Devlin, M. Bosma, G. Mishra, A. Roberts, P. Barham, H. W. Chung, C. Sutton, S. Gehrmann,

Original

2022

Earlier work this paper cites.

J. Liang, W. Huang, F. Xia, P. Xu, K. Hausman, B. Ichter, P. Florence, and A. Zeng, “Code as policies: Language model programs for embodied control,” in

2023

Earlier work this paper cites.

W. Huang, C. Wang, R. Zhang, Y. Li, J. Wu, and L. Fei-Fei, “Voxposer: Composable 3d value maps for robotic manipulation with language models,”

Original

2023

Cited alongside, same era.

L. Guan, K. Valmeekam, S. Sreedharan, and S. Kambhampati, “Leveraging pre-trained large language models to construct and utilize world models for model-based task planning,”

2023

Cited alongside, same era.

K. Valmeekam, S. Sreedharan, M. Marquez, A. Olmo, and S. Kambhampati, “On the planning abilities of large language models (a critical investigation with a proposed benchmark),”

Original

2023

Cited alongside, same era.

J. Achiam, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, F. L. Aleman, D. Almeida, J. Altenschmidt, S. Altman, S. Anadkat,

Original

2023

Cited alongside, same era.

Y. Du, M. Yang, P. Florence, F. Xia, A. Wahid, B. Ichter, P. Sermanet, T. Yu, P. Abbeel, J. B. Tenenbaum,

A. O’Neill, A. Rehman, A. Maddukuri, A. Gupta, A. Padalkar, A. Lee, A. Pooley, A. Gupta, A. Mandlekar, A. Jain,

2024

Closest in time.

B. Li, Y. Zhang, D. Guo, R. Zhang, F. Li, H. Zhang, K. Zhang, Y. Li, Z. Liu, and C. Li, “Llava-onevision: Easy visual task transfer,”

Original

2024

Closest in time.

M. Reid, N. Savinov, D. Teplyashin, D. Lepikhin, T. Lillicrap, J.-b. Alayrac, R. Soricut, A. Lazaridou, O. Firat, J. Schrittwieser,

Original

2024

Closest in time.

T. Silver, S. Dan, K. Srinivas, J. B. Tenenbaum, L. Kaelbling, and M. Katz, “Generalized planning in pddl domains with pretrained large language models,” in

2024

Closest in time.

Z. Michał, C. William, P. Karl, M. Oier, F. Chelsea, and L. Sergey, “Robotic control via embodied chain-of-thought reasoning,”

Original