2023

Text2Motion: From Natural Language Instructions to Feasible Plans

Lin, Kevin, Agia, Christopher, Migimatsu, Toki et al.

Understand

We propose Text2Motion, a language-based planning framework enabling robots to solve sequential manipulation tasks that require long-horizon reasoning.

Given a natural language instruction, our framework constructs both a task- and motion-level plan that is verified to reach inferred symbolic goals.
Text2Motion uses feasibility heuristics encoded in Q-functions of a library of skills to guide task planning with Large Language Models.
Whereas previous language-based planners only consider the feasibility of individual skills, Text2Motion actively resolves geometric dependencies spanning skill sequences by performing geometric feasibility planning during its search.

Built on

Khatib, O.: A unified approach for motion and force control of robot manipulators: The operational space formulation. IEEE Journal on Robotics and Automation 3
1987
Earlier work this paper cites.
Aeronautiques, C., Howe, A., Knoblock, C., McDermott, I.D., Ram, A., Veloso, M., Weld, D., SRI, D.W., Barrett, A., Christianson, D., et al.: Pddl| the planning domain definition language. Technical Report, Tech. Rep. (1998)
1998
Earlier work this paper cites.
Rubinstein, R.: The cross-entropy method for combinatorial and continuous optimization. Methodology and computing in applied probability 1
1999
Earlier work this paper cites.
Bonet, B., Geffner, H.: Planning as heuristic search. Artificial Intelligence 129
2001
Earlier work this paper cites.
Helmert, M.: The fast downward planning system. Journal of Artificial Intelligence Research 26
2006
Earlier work this paper cites.
Kaelbling, L.P., Lozano-Pérez, T.: Hierarchical task and motion planning in the now. In: 2011 IEEE International Conference on Robotics and Automation, pp. 1470–1477 (2011). https://doi.org/10.1109/ICRA.2011.5980391
2011
Earlier work this paper cites.
Kaelbling, L.P., Lozano-Pérez, T.: Hierarchical task and motion planning in the now. In: 2011 IEEE International Conference on Robotics and Automation, pp. 1470–1477 (2011). https://doi.org/10.1109/ICRA.2011.5980391
2011
Earlier work this paper cites.
Kaelbling, L.P., Lozano-Pérez, T.: Integrated robot task and motion planning in the now. Technical report, MASSACHUSETTS INST OF TECH CAMBRIDGE COMPUTER SCIENCE AND ARTIFICIAL … (2012)
2012
Earlier work this paper cites.
Felip, J., Laaksonen, J., Morales, A., Kyrki, V.: Manipulation primitives: A paradigm for abstraction and execution of grasping and manipulation tasks. Robotics and Autonomous Systems 61
2013
Earlier work this paper cites.
Lagriffoul, F., Dimitrov, D., Bidot, J., Saffiotti, A., Karlsson, L.: Efficiently combining task and motion planning using geometric constraints. The International Journal of Robotics Research 33
2014
Earlier work this paper cites.
Toussaint, M.: Logic-geometric programming: An optimization-based approach to combined task and motion planning. In: Twenty-Fourth International Joint Conference on Artificial Intelligence (2015)
2015
Earlier work this paper cites.
Williams, G., Aldrich, A., Theodorou, E.: Model predictive path integral control using covariance variable importance sampling. arXiv preprint arXiv:1509.01149 (2015)
Original
2015
Earlier work this paper cites.
Dantam, N.T., Kingston, Z.K., Chaudhuri, S., Kavraki, L.E.: Incremental task and motion planning: A constraint-based approach. In: Robotics: Science and Systems, AnnArbor, Michigan (2016). https://doi.org/10.15607/RSS.2016.XII.002
2016
Earlier work this paper cites.
Kroemer, O., Sukhatme, G.S.: Learning spatial preconditions of manipulation skills using random forests. In: 2016 IEEE-RAS 16th International Conference on Humanoid Robots (Humanoids), pp. 676–683 (2016). IEEE
2016
Earlier work this paper cites.
Bidot, J., Karlsson, L., Lagriffoul, F., Saffiotti, A.: Geometric backtracking for combined task and motion planning in robotic systems. Artificial Intelligence 247
2017
Earlier work this paper cites.
Lakshminarayanan, B., Pritzel, A., Blundell, C.: Simple and scalable predictive uncertainty estimation using deep ensembles. Advances in neural information processing systems 30
2017
Earlier work this paper cites.
Loshchilov, I., Hutter, F.: SGDR: Stochastic gradient descent with warm restarts. In: International Conference on Learning Representations (2017). https://openreview.net/forum?id=Skq89Scxx
2017
Earlier work this paper cites.
Wang, Z., Garrett, C.R., Kaelbling, L.P., Lozano-Pérez, T.: Active model learning and diverse action sampling for task and motion planning. In: 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 4107–4114 (2018). IEEE
2018
Earlier work this paper cites.
Ames, B., Thackston, A., Konidaris, G.: Learning symbolic representations for planning with parameterized skills. In: 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 526–533 (2018). IEEE
2018
Earlier work this paper cites.
Konidaris, G., Kaelbling, L.P., Lozano-Perez, T.: From skills to symbols: Learning symbolic representations for abstract high-level planning. Journal of Artificial Intelligence Research 61
2018
Earlier work this paper cites.
Haarnoja, T., Zhou, A., Abbeel, P., Levine, S.: Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In: Dy, J., Krause, A. (eds.) Proceedings of the 35th International Conference on Machine Learning. Proceedings of Machine Learning Research, vol. 80, pp. 1861–1870. PMLR, ??? (2018). https://proceedings.mlr.press/v80/haarnoja18b.html
2018
Earlier work this paper cites.
Driess, D., Oguz, O., Toussaint, M.: Hierarchical task and motion planning using logic-geometric programming (hlgp). In: RSS Workshop on Robust Task and Motion Planning (2019)
2019
Earlier work this paper cites.
Kim, B., Kaelbling, L.P., Lozano-Pérez, T.: Adversarial actor-critic method for task and motion planning problems using planning experience. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 33, pp. 8017–8024 (2019)
2019
Earlier work this paper cites.
Stepputtis, S., Campbell, J., Phielipp, M., Lee, S., Baral, C., Ben Amor, H.: Language-conditioned imitation learning for robot manipulation tasks. Advances in Neural Information Processing Systems 33
2020
Earlier work this paper cites.
Garrett, C.R., Lozano-Pérez, T., Kaelbling, L.P.: Pddlstream: Integrating symbolic planners and blackbox samplers via optimistic adaptive planning. In: Proceedings of the International Conference on Automated Planning and Scheduling, vol. 30, pp. 440–448 (2020)
2020
Earlier work this paper cites.
Driess, D., Ha, J.-S., Toussaint, M.: Deep visual reasoning: Learning to predict action sequences for task and motion planning from an initial scene image. arXiv preprint arXiv:2006.05398 (2020)
Original
2020
Earlier work this paper cites.

Similar

Driess, D., Oguz, O., Ha, J.-S., Toussaint, M.: Deep visual heuristics: Learning feasibility of mixed-integer programs for manipulation planning. In: 2020 IEEE International Conference on Robotics and Automation (ICRA), pp. 9563–9569 (2020). IEEE
2020
Cited alongside, same era.
Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J.D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al
2020
Cited alongside, same era.
Zhou, L., Palangi, H., Zhang, L., Hu, H., Corso, J., Gao, J.: Unified vision-language pre-training for image captioning and vqa. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 34, pp. 13041–13049 (2020)
2020
Cited alongside, same era.
Bommasani, R., Hudson, D.A., Adeli, E., Altman, R., Arora, S., Arx, S., Bernstein, M.S., Bohg, J., Bosselut, A., Brunskill, E., et al.: On the opportunities and risks of foundation models. arXiv preprint arXiv:2108.07258 (2021)
Original
2021
Cited alongside, same era.
Jang, E., Irpan, A., Khansari, M., Kappler, D., Ebert, F., Lynch, C., Levine, S., Finn, C.: BC-z: Zero-shot task generalization with robotic imitation learning. In: 5th Annual Conference on Robot Learning (2021). https://openreview.net/forum?id=8kbp23tSGYv
2021
Cited alongside, same era.
Shao, L., Migimatsu, T., Zhang, Q., Yang, K., Bohg, J.: Concept2robot: Learning manipulation concepts from instructions and human demonstrations. The International Journal of Robotics Research 40
2021
Cited alongside, same era.
Garrett, C.R., Chitnis, R., Holladay, R., Kim, B., Silver, T., Kaelbling, L.P., Lozano-Pérez, T.: Integrated task and motion planning. Annual review of control, robotics, and autonomous systems 4
2021
Cited alongside, same era.
Xu, D., Mandlekar, A., Martín-Martín, R., Zhu, Y., Savarese, S., Fei-Fei, L.: Deep affordance foresight: Planning through what can be done in the future. In: 2021 IEEE International Conference on Robotics and Automation (ICRA), pp. 6206–6213 (2021). IEEE
2021
Cited alongside, same era.
Driess, D., Ha, J.-S., Toussaint, M.: Learning to solve sequential physical reasoning problems from a scene image. The International Journal of Robotics Research 40
2021
Cited alongside, same era.
Driess, D., Ha, J.-S., Tedrake, R., Toussaint, M.: Learning geometric reasoning and control for long-horizon tasks from visual input. In: 2021 IEEE International Conference on Robotics and Automation (ICRA), pp. 14298–14305 (2021). IEEE
2021
Cited alongside, same era.
Chitnis, R., Silver, T., Kim, B., Kaelbling, L., Lozano-Perez, T.: Camps: Learning context-specific abstractions for efficient planning in factored mdps. In: Conference on Robot Learning, pp. 64–79 (2021). PMLR
2021
Cited alongside, same era.
Silver, T., Chitnis, R., Curtis, A., Tenenbaum, J.B., Lozano-Perez, T., Kaelbling, L.P.: Planning with learned object importance in large problem instances using graph neural networks. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 35, pp. 11962–11971 (2021)
2021
Cited alongside, same era.

Then

Brohan, A., Brown, N., Carbajal, J., Chebotar, Y., Dabis, J., Finn, C., Gopalakrishnan, K., Hausman, K., Herzog, A., Hsu, J., Ibarz, J., Ichter, B., Irpan, A., Jackson, T., Jesmonth, S., Joshi, N., Julian, R., Kalashnikov, D., Kuang, Y., Leal, I., Lee, K.-H., Levine, S., Lu, Y., Malla, U., Manjunath, D., Mordatch, I., Nachum, O., Parada, C., Peralta, J., Perez, E., Pertsch, K., Quiambao, J., Rao, K., Ryoo, M., Salazar, G., Sanketi, P., Sayed, K., Singh, J., Sontakke, S., Stone, A., Tan, C., Tran, H., Vanhoucke, V., Vega, S., Vuong, Q., Xia, F., Xiao, T., Xu, P., Xu, S., Yu, T., Zitkovich, B.: Rt-1: Robotics transformer for real-world control at scale. In: arXiv Preprint arXiv:2212.06817 (2022)
Original
2022
Later among the works it cites.
Huang, W., Abbeel, P., Pathak, D., Mordatch, I.: Language models as zero-shot planners: Extracting actionable knowledge for embodied agents. arXiv preprint arXiv:2201.07207 (2022)
Original
2022
Later among the works it cites.
Silver, T., Hariprasad, V., Shuttleworth, R.S., Kumar, N., Lozano-Pérez, T., Kaelbling, L.P.: Pddl planning with pretrained large language models. In: NeurIPS 2022 Foundation Models for Decision Making Workshop
2022
Later among the works it cites.
Singh, I., Blukis, V., Mousavian, A., Goyal, A., Xu, D., Tremblay, J., Fox, D., Thomason, J., Garg, A.: Progprompt: Generating situated robot task plans using large language models. arXiv preprint arXiv:2209.11302 (2022)
Original
2022
Later among the works it cites.
Zelikman, E., Huang, Q., Poesia, G., Goodman, N.D., Haber, N.: Parsel: A unified natural language framework for algorithmic reasoning. arXiv preprint arXiv:2212.10561 (2022)
Original
2022
Later among the works it cites.
Kim, B., Shimanuki, L., Kaelbling, L.P., Lozano-Pérez, T.: Representation, learning, and planning algorithms for geometric task and motion planning. The International Journal of Robotics Research 41
2022
Later among the works it cites.
Silver, T., Athalye, A., Tenenbaum, J.B., Lozano-Pérez, T., Kaelbling, L.P.: Learning neuro-symbolic skills for bilevel planning. In: 6th Annual Conference on Robot Learning (2022). https://openreview.net/forum?id=OIaJRUo5UXy
2022
Later among the works it cites.
Curtis, A., Silver, T., Tenenbaum, J.B., Lozano-Pérez, T., Kaelbling, L.: Discovering state and action abstractions for generalized task and motion planning. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 36, pp. 5377–5384 (2022)
2022
Later among the works it cites.
Chitnis, R., Silver, T., Tenenbaum, J.B., Lozano-Perez, T., Kaelbling, L.P.: Learning neuro-symbolic relational transition models for bilevel planning. In: 2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 4166–4173 (2022). IEEE
2022
Later among the works it cites.
Chen, B., Xia, F., Ichter, B., Rao, K., Gopalakrishnan, K., Ryoo, M.S., Stone, A., Kappler, D.: Open-vocabulary queryable scene representations for real world planning. arXiv preprint arXiv:2209.09874 (2022)
Original
2022
Later among the works it cites.
Curtis, A., Fang, X., Kaelbling, L.P., Lozano-Pérez, T., Garrett, C.R.: Long-horizon manipulation of unknown objects via task and motion planning with estimated affordances. In: 2022 International Conference on Robotics and Automation (ICRA), pp. 1940–1946 (2022). IEEE
2022
Later among the works it cites.
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C.L., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al.: Training language models to follow instructions with human feedback. arXiv preprint arXiv:2203.02155 (2022)
Original
2022
Later among the works it cites.
Li, X.L., Holtzman, A., Fried, D., Liang, P., Eisner, J., Hashimoto, T., Zettlemoyer, L., Lewis, M.: Contrastive decoding: Open-ended text generation as optimization. arXiv preprint arXiv:2210.15097 (2022)
Original
2022
Later among the works it cites.
Chen, Y., Yuan, L., Cui, G., Liu, Z., Ji, H.: A close look into the calibration of pre-trained language models. arXiv preprint arXiv:2211.00151 (2022)
Original
2022
Later among the works it cites.
Zeng, A., Wong, A., Welker, S., Choromanski, K., Tombari, F., Purohit, A., Ryoo, M., Sindhwani, V., Lee, J., Vanhoucke, V., et al.: Socratic models: Composing zero-shot multimodal reasoning with language. arXiv preprint arXiv:2204.00598 (2022)
Original
2022
Later among the works it cites.
Wu, J., Antonova, R., Kan, A., Lepert, M., Zeng, A., Song, S., Bohg, J., Rusinkiewicz, S., Funkhouser, T.: Tidybot: Personalized robot assistance with large language models. arXiv preprint arXiv:2305.05658 (2023)
Original
2023
Closest in time.
Dalal, M., Mandlekar, A., Garrett, C., Handa, A., Salakhutdinov, R., Fox, D.: Imitating task and motion planning with visuomotor transformers. arXiv preprint arXiv:2305.16309 (2023)
Original
2023
Closest in time.
Liu, B., Jiang, Y., Zhang, X., Liu, Q., Zhang, S., Biswas, J., Stone, P.: Llm+ p: Empowering large language models with optimal planning proficiency. arXiv preprint arXiv:2304.11477 (2023)
Original
2023
Closest in time.
Wang, Z., Cai, S., Liu, A., Ma, X., Liang, Y.: Describe, explain, plan and select: Interactive planning with large language models enables open-world multi-task agents. arXiv preprint arXiv:2302.01560 (2023)
Original
2023
Closest in time.
Skreta, M., Yoshikawa, N., Arellano-Rubach, S., Ji, Z., Kristensen, L.B., Darvish, K., Aspuru-Guzik, A., Shkurti, F., Garg, A.: Errors are useful prompts: Instruction guided task programming with verifier-assisted iterative prompting. arXiv preprint arXiv:2303.14100 (2023)
Original
2023
Closest in time.
Vemprala, S., Bonatti, R., Bucker, A., Kapoor, A.: Chatgpt for robotics: Design principles and model abilities. Technical Report MSR-TR-2023-8, Microsoft (February 2023)
2023
Closest in time.
Driess, D., Huang, Z., Li, Y., Tedrake, R., Toussaint, M.: Learning multi-object dynamics with compositional neural radiance fields. In: Proceedings of The 6th Conference on Robot Learning. Proceedings of Machine Learning Research, vol. 205, pp. 1755–1768. PMLR, ??? (2023)
2023
Closest in time.
Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., et al.: Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971 (2023)
Original
2023
Closest in time.
Driess, D., Xia, F., Sajjadi, M.S., Lynch, C., Chowdhery, A., Ichter, B., Wahid, A., Tompson, J., Vuong, Q., Yu, T., et al.: Palm-e: An embodied multimodal language model. arXiv preprint arXiv:2303.03378 (2023)
Original
2023
Closest in time.
OpenAI: GPT-4 Technical Report (2023)
2023
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…