GPT-4V(ision) for Robotics: Multimodal Task Planning from Human Demonstration

H. T. Kuhn and W. L. Inequalities, “Related systems,” Annals of Mathematic Studies, Princeton Univ. Press. EEUU

1956

Earlier work this paper cites.

T. Yoshikawa, “Passive and active closures by constraining mechanisms,” in 1996 ICRA

1996

Earlier work this paper cites.

Psychology press, 2014

J. J. Gibson, The ecological approach to visual perception: classic edition · 2014

Earlier work this paper cites.

R. Goyal, S. Ebrahimi Kahou, V. Michalski, J. Materzynska, S. Westphal, H. Kim, V. Haenel, I. Fruend, P. Yianilos, M. Mueller-Freitag, et al

2017

Earlier work this paper cites.

A. Saudabayev, Z. Rysbek, R. Khassenova, and H. A. Varol, “Human grasping database for activities of daily living with depth, color and kinematic data streams,” Scientific data

2018

Earlier work this paper cites.

P. Pramanick, H. B. Barua, and C. Sarkar, “Decomplex: Task planning from complex natural instructions by a collocating robot,” in 2020 IROS

2020

Earlier work this paper cites.

C. Lynch and P. Sermanet, “Language conditioned imitation learning over unstructured data,” arXiv preprint arXiv:2005.07648

Original

2020

Earlier work this paper cites.

K. Sasabuchi, N. Wake, and K. Ikeuchi, “Task-oriented motion mapping on robots of various configuration using body role division,” IEEE Robotics and Automation Letters

2020

Earlier work this paper cites.

D. Damen, H. Doughty, G. M. Farinella, S. Fidler, A. Furnari, E. Kazakos, D. Moltisanti, J. Munro, T. Perrett, W. Price, et al

2020

Earlier work this paper cites.

S. G. Venkatesh, R. Upadrashta, and B. Amrutur, “Translating natural language instructions to computer programs for robot manipulation,” in 2021 IROS

2021

Earlier work this paper cites.

C. R. Garrett, R. Chitnis, R. Holladay, B. Kim, T. Silver, L. P. Kaelbling, and T. Lozano-Pérez, “Integrated task and motion planning,” Annual review of control, robotics, and autonomous systems

2021

Earlier work this paper cites.

N. Wake, R. Arakawa, I. Yanokura, T. Kiyokawa, K. Sasabuchi, J. Takamatsu, and K. Ikeuchi, “A learning-from-observation framework: One-shot robot teaching for grasp-manipulation-release household operations,” in 2021 SII

2021

Earlier work this paper cites.

N. Wake, I. Yanokura, K. Sasabuchi, and K. Ikeuchi, “Verbal focus-of-attention system for learning-from-demonstration,” in 2021 ICRA

2021

Earlier work this paper cites.

Y. Jiang, A. Gupta, Z. Zhang, G. Wang, Y. Dou, Y. Chen, L. Fei-Fei, A. Anandkumar, Y. Zhu, and L. Fan, “Vima: General robot manipulation with multimodal prompts,” arXiv

2022

Earlier work this paper cites.

A. Brohan, N. Brown, J. Carbajal, Y. Chebotar, J. Dabis, C. Finn, K. Gopalakrishnan, K. Hausman, A. Herzog, J. Hsu, et al

Original

2022

Earlier work this paper cites.

M. Ahn, A. Brohan, N. Brown, Y. Chebotar, O. Cortes, B. David, C. Finn, K. Gopalakrishnan, K. Hausman, A. Herzog, et al

Original

2022

Earlier work this paper cites.

W. Huang, P. Abbeel, D. Pathak, and I. Mordatch, “Language models as zero-shot planners: Extracting actionable knowledge for embodied agents,” in 2022 ICML

2022

Earlier work this paper cites.

I. Yanaokura, N. Wake, K. Sasabuchi, R. Arakawa, K. Okada, J. Takamatsu, M. Inaba, and K. Ikeuchi, “A multimodal learning-from-observation towards all-at-once robot teaching using task cohesion,” in 2022 SII

2022

Earlier work this paper cites.

W. Huang, F. Xia, T. Xiao, H. Chan, J. Liang, P. Florence, A. Zeng, J. Tompson, I. Mordatch, Y. Chebotar, et al

Original

2022

Earlier work this paper cites.

Y. Ding, X. Zhang, S. Amiri, N. Cao, H. Yang, C. Esselink, and S. Zhang, “Robot task planning and situation handling in open worlds,” arXiv preprint arXiv:2210.01287

Original

2022

Earlier work this paper cites.

A. Zeng, M. Attarian, B. Ichter, K. Choromanski, A. Wong, S. Welker, F. Tombari, A. Purohit, M. Ryoo, V. Sindhwani, et al

Original

2022

Earlier work this paper cites.

S. S. Raman, V. Cohen, E. Rosen, I. Idrees, D. Paulius, and S. Tellex, “Planning with large language models via corrective re-prompting,” arXiv preprint arXiv:2211.09935

Original

2022

Earlier work this paper cites.

H. Liu, L. Lee, K. Lee, and P. Abbeel, “Instruction-following agents with multimodal transformer,” arXiv preprint arXiv:2210.13431

Original

2022

Earlier work this paper cites.

X. Zhou, R. Girdhar, A. Joulin, P. Krähenbühl, and I. Misra, “Detecting twenty-thousand classes using image-level supervision,” in 2022 ECCV

2022

Cited alongside, same era.

D. Saito, K. Sasabuchi, N. Wake, J. Takamatsu, H. Koike, and K. Ikeuchi, “Task-grasping from a demonstrated human strategy,” in 2022 Humanoids

2022

Cited alongside, same era.

J. Takamatsu, K. Sasabuchi, N. Wake, A. Kanehira, and K. Ikeuchi, “Learning-from-observation system considering hardware-level reusability,” arXiv preprint arXiv:2212.09242

Original

2022

Cited alongside, same era.

A. Brohan, N. Brown, J. Carbajal, Y. Chebotar, X. Chen, K. Choromanski, T. Ding, D. Driess, A. Dubey, C. Finn, et al

Original

2023

Cited alongside, same era.

X. Li, M. Liu, H. Zhang, C. Yu, J. Xu, H. Wu, C. Cheang, Y. Jing, W. Zhang, H. Liu, et al

J. Pan, G. Chou, and D. Berenson, “Data-efficient learning of natural language to linear temporal logic translators for robot task specification,” arXiv preprint arXiv:2303.08006

Original

2023

Closest in time.

K. Lin, C. Agia, T. Migimatsu, M. Pavone, and J. Bohg, “Text2motion: From natural language instructions to feasible plans,” arXiv preprint arXiv:2303.12153

Original

2023

Closest in time.

C. Zhao, S. Yuan, C. Jiang, J. Cai, H. Yu, M. Y. Wang, and Q. Chen, “Erra: An embodied representation and reasoning architecture for long-horizon language-conditioned manipulation tasks,” IEEE Robotics and Automation Letters

2023

Closest in time.

O. Mees, J. Borja-Diaz, and W. Burgard, “Grounding language with visual affordances over unstructured data,” in 2023 ICRA

2023

Closest in time.

J. Sun, Q. Zhang, Y. Duan, X. Jiang, C. Cheng, and R. Xu, “Prompt, plan, perform: Llm-based humanoid control via quantized imitation learning,” arXiv preprint arXiv:2309.11359