Fetching the paper…

Reading the bibliography…

2024

NARRATE: Versatile Language Architecture for Optimal Control in Robotics

Ismail, Seif, Arbues, Antonio, Cotterell, Ryan et al.

Original
PDF
Post on X

Understand

The impressive capabilities of Large Language Models (LLMs) have led to various efforts to enable robots to be controlled through natural language instructions, opening exciting possibilities for human-robot interaction The goal is for the motor-control task to be performed accurately, efficiently and safely while also enjoying the flexibility imparted by LLMs to specify and adjust the task through natural language.

In this work, we demonstrate how a careful layering of an LLM in combination with a Model Predictive Control (MPC) formulation allows for accurate and flexible robotic control via natural language while taking into consideration safety constraints.
In particular, we rely on the LLM to effectively frame constraints and objective functions as mathematical expressions, which are later used in the motor-control module via MPC.
The transparency of the optimization formulation allows for interpretability of the task and enables adjustments through human feedback.

NARRATE: Versatile Language Architecture for Optimal Control in Robotics · Around

Built on

H. Baruh,
1998
Earlier work this paper cites.
D. T. Coleman, I. A. Sucan, S. Chitta, and N. Correll, “Reducing the barrier to entry of complex robotic software: a moveit! case study,” 2014. [Online]. Available:
2014
Earlier work this paper cites.
J. B. Rawlings, D. Q. Mayne, and M. Diehl,
2017
Earlier work this paper cites.
P. Goyal, S. Niekum, and R. J. Mooney, “Using natural language for reward shaping in reinforcement learning,”
Original
2019
Earlier work this paper cites.
J. A. E. Andersson, J. Gillis, G. Horn, J. B. Rawlings, and M. Diehl, “CasADi – A software framework for nonlinear optimization and optimal control,”
2019
Earlier work this paper cites.
J. Austin, A. Odena, M. Nye, M. Bosma, H. Michalewski, D. Dohan, E. Jiang, C. Cai, M. Terry, Q. Le,
Original
2021
Earlier work this paper cites.
D. Trivedi, J. Zhang, S.-H. Sun, and J. J. Lim, “Learning to synthesize programs as interpretable and generalizable policies,”
2021
Earlier work this paper cites.
S. Arora and P. Doshi, “A survey of inverse reinforcement learning: Challenges, methods and progress,”
2021
Earlier work this paper cites.
Q. Gallouédec, N. Cazin, E. Dellandréa, and L. Chen, “panda-gym: Open-Source Goal-Conditioned Environments for Robotic Learning,”
2021
Earlier work this paper cites.
Y. Jiang, A. Gupta, Z. Zhang, G. Wang, Y. Dou, Y. Chen, L. Fei-Fei, A. Anandkumar, Y. Zhu, and L. Fan, “Vima: General robot manipulation with multimodal prompts,”
Original
2022
Earlier work this paper cites.
M. Shridhar, L. Manuelli, and D. Fox, “Perceiver-actor: A multi-task transformer for robotic manipulation,” in
2022
Earlier work this paper cites.

Similar

Y. Li, D. Choi, J. Chung, N. Kushman, J. Schrittwieser, R. Leblond, T. Eccles, J. Keeling, F. Gimeno, A. Dal Lago,
2022
Cited alongside, same era.
P. Sharma, B. Sundaralingam, V. Blukis, C. Paxton, T. Hermans, A. Torralba, J. Andreas, and D. Fox, “Correcting robot plans with natural language feedback,”
Original
2022
Cited alongside, same era.
A. Suhr and Y. Artzi, “Continual learning for instruction following from realtime feedback,”
Original
2022
Cited alongside, same era.
K. Hausman, B. Ichter, S. Levine, A. Toshev, F. Xia, and C. Parada, “Natural language control of a robot,” Oct. 5 2023, uS Patent App. 18/128,953
2023
Cited alongside, same era.
D. Shah, M. Equi, B. Osinski, F. Xia, B. Ichter, and S. Levine, “Navigation with large language models: Semantic guesswork as a heuristic for planning,”
Original

Then

Y. Wang, Z. Xian, F. Chen, T.-H. Wang, Y. Wang, Z. Erickson, D. Held, and C. Gan, “Robogen: Towards unleashing infinite data for automated robot learning via generative simulation,” 2023
2023
Later among the works it cites.
Y.-J. Wang, B. Zhang, J. Chen, and K. Sreenath, “Prompt a robot to walk with large language models,” 2023
2023
Later among the works it cites.
J. Liang, W. Huang, F. Xia, P. Xu, K. Hausman, B. Ichter, P. Florence, and A. Zeng, “Code as policies: Language model programs for embodied control,” in
2023
Later among the works it cites.
W. Yu, N. Gileadi, C. Fu, S. Kirmani, K.-H. Lee, M. G. Arenas, H.-T. L. Chiang, T. Erez, L. Hasenclever, J. Humplik,
Original
2023
Later among the works it cites.
C. Wu, S. Yin, W. Qi, X. Wang, Z. Tang, and N. Duan, “Visual chatgpt: Talking, drawing and editing with visual foundation models,”

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

2023

Cited alongside, same era.

B. M. Lake and M. Baroni, “Human-like systematic generalization through a meta-learning neural network,”

2023

Cited alongside, same era.

I. Singh, V. Blukis, A. Mousavian, A. Goyal, D. Xu, J. Tremblay, D. Fox, J. Thomason, and A. Garg, “Progprompt: Generating situated robot task plans using large language models,” in

2023

Cited alongside, same era.

S. Vemprala, R. Bonatti, A. Bucker, and A. Kapoor, “Chatgpt for robotics: Design principles and model abilities,”

2023

Cited alongside, same era.

A. B. et al., “Rt-2: Vision-language-action models transfer web knowledge to robotic control,” 2023

2023

Cited alongside, same era.

O. X.-E. Collaboration, “Open X-Embodiment: Robotic learning datasets and RT-X models,”

2023

Cited alongside, same era.

K. B. et al., “Robocat: A self-improving generalist agent for robotic manipulation,” 2023

2023

Cited alongside, same era.

F. Farshidian

Cited in the paper.

2023

Later among the works it cites.

Z. Mandi, S. Jain, and S. Song, “Roco: Dialectic multi-robot collaboration with large language models,”

2023

Later among the works it cites.

G. Wang, Y. Xie, Y. Jiang, A. Mandlekar, C. Xiao, Y. Zhu, L. Fan, and A. Anandkumar, “Voyager: An open-ended embodied agent with large language models,”

2023

Later among the works it cites.

OpenAI, “Gpt-4 technical report,”

2023

Later among the works it cites.

Y. J. Ma, W. Liang, G. Wang, D.-A. Huang, O. Bastani, D. Jayaraman, Y. Zhu, L. Fan, and A. Anandkumar, “Eureka: Human-level reward design via coding large language models,”

2023

Later among the works it cites.

W. Huang, C. Wang, R. Zhang, Y. Li, J. Wu, and L. Fei-Fei, “Voxposer: Composable 3d value maps for robotic manipulation with language models,”

2023

Later among the works it cites.

Z. Fu, T. Z. Zhao, and C. Finn, “Mobile aloha: Learning bimanual mobile manipulation with low-cost whole-body teleoperation,” 2024

2024

Closest in time.