DoReMi: Grounding Language Model by Detecting and Recovering from Plan-Execution Misalignment

R. E. Fikes and N. J. Nilsson, “Strips: A new approach to the application of theorem proving to problem solving,” Artificial intelligence , vol. 2, no. 3-4, pp. 189–208, 1971

1971

Earlier work this paper cites.

D. Nau, Y. Cao, A. Lotem, and H. Munoz-Avila, “Shop: Simple hierarchical ordered planner,” in Proceedings of the 16th international joint conference on Artificial intelligence-Volume 2 , 1999, pp. 968–973

1999

Earlier work this paper cites.

M. Fox and D. Long, “Pddl2. 1: An extension to pddl for expressing temporal planning domains,” Journal of artificial intelligence research , vol. 20, pp. 61–124, 2003

2003

Earlier work this paper cites.

M. MacMahon, B. Stankiewicz, and B. Kuipers, “Walk the talk: Connecting language, knowledge, and action in route instructions,” Def , vol. 2, no. 6, p. 4, 2006

2006

Earlier work this paper cites.

S. Antol, A. Agrawal, J. Lu, M. Mitchell, D. Batra, C. L. Zitnick, and D. Parikh, “Vqa: Visual question answering,” in Proceedings of the IEEE international conference on computer vision , 2015, pp. 2425–2433

2015

Earlier work this paper cites.

H. Mei, M. Bansal, and M. Walter, “Listen, attend, and walk: Neural mapping of navigational instructions to action sequences,” in Proceedings of the AAAI Conference on Artificial Intelligence , vol. 30, no. 1, 2016

2016

Earlier work this paper cites.

D. Misra, J. Langford, and Y. Artzi, “Mapping instructions and visual observations to actions with reinforcement learning,” arXiv preprint arXiv:1704.08795 , 2017

Original

2017

Earlier work this paper cites.

D. S. Chaplot, K. M. Sathyendra, R. K. Pasumarthi, D. Rajagopal, and R. Salakhutdinov, “Gated-attention architectures for task-oriented language grounding,” in Proceedings of the AAAI Conference on Artificial Intelligence , vol. 32, no. 1, 2018

2018

Earlier work this paper cites.

X. Puig, K. Ra, M. Boben, J. Li, T. Wang, S. Fidler, and A. Torralba, “Virtualhome: Simulating household activities via programs,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , 2018, pp. 8494–8502

2018

Earlier work this paper cites.

X. B. Peng, P. Abbeel, S. Levine, and M. Van de Panne, “Deepmimic: Example-guided deep reinforcement learning of physics-based character skills,” ACM Transactions On Graphics (TOG) , vol. 37, no. 4, pp. 1–14, 2018

2018

Earlier work this paper cites.

Y. Jiang, S. S. Gu, K. P. Murphy, and C. Finn, “Language as an abstraction for hierarchical deep reinforcement learning,” Advances in Neural Information Processing Systems , vol. 32, 2019

2019

Earlier work this paper cites.

Y.-q. Jiang, S.-q. Zhang, P. Khandelwal, and P. Stone, “Task planning in robotics: an empirical comparison of pddl-and asp-based systems,” Frontiers of Information Technology & Electronic Engineering , vol. 20, pp. 363–373, 2019

2019

Earlier work this paper cites.

M. Shridhar, J. Thomason, D. Gordon, Y. Bisk, W. Han, R. Mottaghi, L. Zettlemoyer, and D. Fox, “Alfred: A benchmark for interpreting grounded instructions for everyday tasks,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , 2020, pp. 10 740–10 749

2020

Earlier work this paper cites.

L. Zhou, H. Palangi, L. Zhang, H. Hu, J. Corso, and J. Gao, “Unified vision-language pre-training for image captioning and vqa,” in Proceedings of the AAAI conference on artificial intelligence , vol. 34, no. 07, 2020, pp. 13 041–13 049

2020

Earlier work this paper cites.

A. Zeng, P. Florence, J. Tompson, S. Welker, J. Chien, M. Attarian, T. Armstrong, I. Krasin, D. Duong, V. Sindhwani, and J. Lee, “Transporter networks: Rearranging the visual world for robotic manipulation,” Conference on Robot Learning (CoRL) , 2020

2020

Earlier work this paper cites.

A. Silva, N. Moorman, W. Silva, Z. Zaidi, N. Gopalan, and M. Gombolay, “Lancon-learn: Learning with language to enable generalization in multi-task manipulation,” IEEE Robotics and Automation Letters , vol. 7, no. 2, pp. 1635–1642, 2021

2021

Earlier work this paper cites.

P. Goyal, S. Niekum, and R. Mooney, “Pixl2r: Guiding reinforcement learning using natural language by mapping pixels to rewards,” in Conference on Robot Learning . PMLR, 2021, pp. 485–497

2021

Cited alongside, same era.

Y. Zhang and J. Chai, “Hierarchical task learning from language instructions with unified transformers and self-monitoring,” arXiv preprint arXiv:2106.03427 , 2021

Original

2021

Cited alongside, same era.

X. Gu, T.-Y. Lin, W. Kuo, and Y. Cui, “Open-vocabulary object detection via vision and language knowledge distillation,” arXiv preprint arXiv:2104.13921 , 2021

Original

2021

Cited alongside, same era.

A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, et al. , “Learning transferable visual models from natural language supervision,” in International conference on machine learning . PMLR, 2021, pp. 8748–8763

2021

Cited alongside, same era.

J. Li, D. Li, C. Xiong, and S. Hoi, “Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation,” in International Conference on Machine Learning . PMLR, 2022, pp. 12 888–12 900

2022

Later among the works it cites.

Y. Ma, F. Farshidian, T. Miki, J. Lee, and M. Hutter, “Combining learning-based locomotion policy with model-based manipulation for legged mobile manipulators,” IEEE Robotics and Automation Letters , vol. 7, no. 2, pp. 2377–2384, 2022

2022

Later among the works it cites.

A. Brohan, N. Brown, J. Carbajal, Y. Chebotar, X. Chen, K. Choromanski, T. Ding, D. Driess, A. Dubey, C. Finn, et al. , “Rt-2: Vision-language-action models transfer web knowledge to robotic control,” arXiv preprint arXiv:2307.15818 , 2023

Original

2023

Closest in time.

M. Shridhar, L. Manuelli, and D. Fox, “Perceiver-actor: A multi-task transformer for robotic manipulation,” in Conference on Robot Learning . PMLR, 2023, pp. 785–799

2023

Closest in time.