Built on

T. Winograd, “Procedures as a representation for data in a computer program for understanding natural language,”
1971
Earlier work this paper cites.
M. MacMahon, B. Stankiewicz, and B. Kuipers, “Walk the talk: Connecting language, knowledge, and action in route instructions,”
2006
Earlier work this paper cites.
J. Dzifcak, M. Scheutz, C. Baral, and P. Schermerhorn, “What to do and how to do it: Translating natural language directives into temporal and dynamic logic representation for goal management and action execution,” in
2009
Earlier work this paper cites.
T. Kollar, S. Tellex, D. Roy, and N. Roy, “Toward understanding natural language directions,” in
2010
Earlier work this paper cites.
S. Tellex, T. Kollar, S. Dickerson, M. Walter, A. Banerjee, S. Teller, and N. Roy, “Understanding natural language commands for robotic navigation and mobile manipulation,” in
2011
Earlier work this paper cites.
Y. Artzi and L. Zettlemoyer, “Weakly supervised learning of semantic parsers for mapping instructions to actions,”
2013
Earlier work this paper cites.
C. Matuszek, E. Herbst, L. Zettlemoyer, and D. Fox, “Learning to parse natural language commands to a robot control system,” in
2013
Earlier work this paper cites.
J. Thomason, S. Zhang, R. J. Mooney, and P. Stone, “Learning to interpret natural language commands through human-robot dialog,” in
2015
Earlier work this paper cites.
C. Breazeal, K. Dautenhahn, and T. Kanda, “Social robotics,”
2016
Earlier work this paper cites.
J. Andreas, D. Klein, and S. Levine, “Learning with latent language,”
Original
2017
Earlier work this paper cites.
D. Misra, J. Langford, and Y. Artzi, “Mapping instructions and visual observations to actions with reinforcement learning,”
Original
2017
Earlier work this paper cites.
D. Kalashnikov, A. Irpan, P. Pastor, J. Ibarz, A. Herzog, E. Jang, D. Quillen, E. Holly, M. Kalakrishnan, V. Vanhoucke
2018
Earlier work this paper cites.
A. Zeng, “Learning visual affordances for robotic manipulation,” Ph.D. dissertation, Princeton University, 2019
2019
Earlier work this paper cites.
J. Luketina, N. Nardelli, G. Farquhar, J. N. Foerster, J. Andreas, E. Grefenstette, S. Whiteson, and T. Rocktäschel, “A survey of reinforcement learning informed by natural language,” in
2019
Earlier work this paper cites.
Y. Jiang, S. S. Gu, K. P. Murphy, and C. Finn, “Language as an abstraction for hierarchical deep reinforcement learning,”
2019
Earlier work this paper cites.
G. Cideron, M. Seurin, F. Strub, and O. Pietquin, “Self-educated language agent with hindsight experience replay for instruction following,”
2019
Earlier work this paper cites.
S. Tellex, N. Gopalan, H. Kress-Gazit, and C. Matuszek, “Robots that use language,”
2020
Earlier work this paper cites.
C. Lynch and P. Sermanet, “Language conditioned imitation learning over unstructured data,”
Original
2020
Earlier work this paper cites.
T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell
2020
Earlier work this paper cites.
D. Hupkes, V. Dankers, M. Mul, and E. Bruni, “Compositionality decomposed: How do neural networks generalise?”
2020
Earlier work this paper cites.

Similar

J. Thomason, A. Padmakumar, J. Sinapov, N. Walker, Y. Jiang, H. Yedidsion, J. Hart, P. Stone, and R. Mooney, “Jointly improving parsing and perception for natural language commands through human-robot dialog,”
2020
Cited alongside, same era.
S. Stepputtis, J. Campbell, M. Phielipp, S. Lee, C. Baral, and H. Ben Amor, “Language-conditioned imitation learning for robot manipulation tasks,”
2020
Cited alongside, same era.
P. Goyal, S. Niekum, and R. J. Mooney, “Pixl2r: Guiding reinforcement learning using natural language by mapping pixels to rewards,”
Original
2020
Cited alongside, same era.
A. Akakzia, C. Colas, P.-Y. Oudeyer, M. Chetouani, and O. Sigaud, “Grounding language to autonomously-acquired skills via goal generation,”
Original
2020
Cited alongside, same era.
K. Ellis, C. Wong, M. Nye, M. Sable-Meyer, L. Cary, L. Morales, L. Hewitt, A. Solar-Lezama, and J. B. Tenenbaum, “Dreamcoder: Growing generalizable, interpretable knowledge with wake-sleep bayesian program learning,”

Then

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…

2022

Code as Policies: Language Model Programs for Embodied Control

Liang, Jacky, Huang, Wenlong, Xia, Fei et al.

Understand

Large language models (LLMs) trained on code completion have been shown to be capable of synthesizing simple Python programs from docstrings [1].

We find that these code-writing LLMs can be re-purposed to write robot policy code, given natural language commands.
Specifically, policy code can express functions or feedback loops that process perception outputs (e.g.,from object detectors [2], [3]) and parameterize control primitive APIs.
When provided as input several example language commands (formatted as comments) followed by corresponding policy code (via few-shot prompting), LLMs can take in new commands and autonomously re-compose API calls to generate new policy code respectively.

Built on

T. Winograd, “Procedures as a representation for data in a computer program for understanding natural language,”
1971
Earlier work this paper cites.
M. MacMahon, B. Stankiewicz, and B. Kuipers, “Walk the talk: Connecting language, knowledge, and action in route instructions,”
2006
Earlier work this paper cites.
J. Dzifcak, M. Scheutz, C. Baral, and P. Schermerhorn, “What to do and how to do it: Translating natural language directives into temporal and dynamic logic representation for goal management and action execution,” in
2009
Earlier work this paper cites.
T. Kollar, S. Tellex, D. Roy, and N. Roy, “Toward understanding natural language directions,” in
2010
Earlier work this paper cites.
S. Tellex, T. Kollar, S. Dickerson, M. Walter, A. Banerjee, S. Teller, and N. Roy, “Understanding natural language commands for robotic navigation and mobile manipulation,” in
2011
Earlier work this paper cites.
Y. Artzi and L. Zettlemoyer, “Weakly supervised learning of semantic parsers for mapping instructions to actions,”
2013
Earlier work this paper cites.
C. Matuszek, E. Herbst, L. Zettlemoyer, and D. Fox, “Learning to parse natural language commands to a robot control system,” in
2013
Earlier work this paper cites.
J. Thomason, S. Zhang, R. J. Mooney, and P. Stone, “Learning to interpret natural language commands through human-robot dialog,” in
2015
Earlier work this paper cites.
C. Breazeal, K. Dautenhahn, and T. Kanda, “Social robotics,”
2016
Earlier work this paper cites.
J. Andreas, D. Klein, and S. Levine, “Learning with latent language,”
Original
2017
Earlier work this paper cites.
D. Misra, J. Langford, and Y. Artzi, “Mapping instructions and visual observations to actions with reinforcement learning,”
Original
2017
Earlier work this paper cites.
D. Kalashnikov, A. Irpan, P. Pastor, J. Ibarz, A. Herzog, E. Jang, D. Quillen, E. Holly, M. Kalakrishnan, V. Vanhoucke
2018
Earlier work this paper cites.
A. Zeng, “Learning visual affordances for robotic manipulation,” Ph.D. dissertation, Princeton University, 2019
2019
Earlier work this paper cites.
J. Luketina, N. Nardelli, G. Farquhar, J. N. Foerster, J. Andreas, E. Grefenstette, S. Whiteson, and T. Rocktäschel, “A survey of reinforcement learning informed by natural language,” in
2019
Earlier work this paper cites.
Y. Jiang, S. S. Gu, K. P. Murphy, and C. Finn, “Language as an abstraction for hierarchical deep reinforcement learning,”
2019
Earlier work this paper cites.
G. Cideron, M. Seurin, F. Strub, and O. Pietquin, “Self-educated language agent with hindsight experience replay for instruction following,”
2019
Earlier work this paper cites.
S. Tellex, N. Gopalan, H. Kress-Gazit, and C. Matuszek, “Robots that use language,”
2020
Earlier work this paper cites.
C. Lynch and P. Sermanet, “Language conditioned imitation learning over unstructured data,”
Original
2020
Earlier work this paper cites.
T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell
2020
Earlier work this paper cites.
D. Hupkes, V. Dankers, M. Mul, and E. Bruni, “Compositionality decomposed: How do neural networks generalise?”
2020
Earlier work this paper cites.

Similar

J. Thomason, A. Padmakumar, J. Sinapov, N. Walker, Y. Jiang, H. Yedidsion, J. Hart, P. Stone, and R. Mooney, “Jointly improving parsing and perception for natural language commands through human-robot dialog,”
2020
Cited alongside, same era.
S. Stepputtis, J. Campbell, M. Phielipp, S. Lee, C. Baral, and H. Ben Amor, “Language-conditioned imitation learning for robot manipulation tasks,”
2020
Cited alongside, same era.
P. Goyal, S. Niekum, and R. J. Mooney, “Pixl2r: Guiding reinforcement learning using natural language by mapping pixels to rewards,”
Original
2020
Cited alongside, same era.
A. Akakzia, C. Colas, P.-Y. Oudeyer, M. Chetouani, and O. Sigaud, “Grounding language to autonomously-acquired skills via goal generation,”
Original
2020
Cited alongside, same era.
K. Ellis, C. Wong, M. Nye, M. Sable-Meyer, L. Cary, L. Morales, L. Hewitt, A. Solar-Lezama, and J. B. Tenenbaum, “Dreamcoder: Growing generalizable, interpretable knowledge with wake-sleep bayesian program learning,”

Then

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Original

2020

Cited alongside, same era.

L. Tian, K. Ellis, M. Kryven, and J. Tenenbaum, “Learning abstract structure for drawing by efficient motor program induction,”

2020

Cited alongside, same era.

O. Mees and W. Burgard, “Composing pick-and-place tasks by grounding language,” in

2020

Cited alongside, same era.

M. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. d. O. Pinto, J. Kaplan, H. Edwards, Y. Burda, N. Joseph, G. Brockman

Original

2021

Cited alongside, same era.

A. Kamath, M. Singh, Y. LeCun, G. Synnaeve, I. Misra, and N. Carion, “Mdetr-modulated detection for end-to-end multi-modal understanding,” in

2021

Cited alongside, same era.

X. Gu, T.-Y. Lin, W. Kuo, and Y. Cui, “Open-vocabulary object detection via vision and language knowledge distillation,”

Original

2021

Cited alongside, same era.

M. Shridhar, L. Manuelli, and D. Fox, “Cliport: What and where pathways for robotic manipulation,” in

2021

Cited alongside, same era.

K. Cobbe, V. Kosaraju, M. Bavarian, J. Hilton, R. Nakano, C. Hesse, and J. Schulman, “Training verifiers to solve math word problems,”

Original

2021

Cited alongside, same era.

P. Florence, C. Lynch, A. Zeng, O. A. Ramirez, A. Wahid, L. Downs, A. Wong, J. Lee, I. Mordatch, and J. Tompson, “Implicit behavioral cloning,” in

2022

Closest in time.

L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. L. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray

Original

2022

Closest in time.

D. Shah, B. Osinski, B. Ichter, and S. Levine, “Lm-nav: Robotic navigation with large pre-trained models of language, vision, and action,”

Original

2022

Closest in time.

S. Nair, E. Mitchell, K. Chen, S. Savarese, C. Finn

2022

Closest in time.

P. Sharma, B. Sundaralingam, V. Blukis, C. Paxton, T. Hermans, A. Torralba, J. Andreas, and D. Fox, “Correcting robot plans with natural language feedback,”

Original

2022

Closest in time.

I. Drori, S. Zhang, R. Shuttleworth, L. Tang, A. Lu, E. Ke, K. Liu, L. Chen, S. Tran, N. Cheng

2022

Closest in time.

A. Lewkowycz, A. Andreassen, D. Dohan, E. Dyer, H. Michalewski, V. Ramasesh, A. Slone, C. Anil, I. Schlag, T. Gutman-Solo

Original

2022

Closest in time.

D. Zhou, N. Schärli, L. Hou, J. Wei, N. Scales, X. Wang, D. Schuurmans, O. Bousquet, Q. Le, and E. Chi, “Least-to-most prompting enables complex reasoning in large language models,”

Original

2022

Closest in time.

J. Wei, X. Wang, D. Schuurmans, M. Bosma, B. Ichter, F. Xia, E. Chi, Q. Le, and D. Zhou, “Chain of thought prompting elicits reasoning in large language models,”

Original

2022

Closest in time.

W. Liu, C. Paxton, T. Hermans, and D. Fox, “Structformer: Learning spatial structure for language-guided semantic rearrangement of novel objects,” in

2022

Closest in time.

W. Yuan, C. Paxton, K. Desingh, and D. Fox, “Sornet: Spatial object-centric representations for sequential manipulation,” in

2022

Closest in time.

A. Bucker, L. Figueredo, S. Haddadin, A. Kapoor, S. Ma, and R. Bonatti, “Reshaping robot trajectories using natural language commands: A study of multi-modal data alignment using transformers,”

Original

2022

Closest in time.

A. Bobu, C. Paxton, W. Yang, B. Sundaralingam, Y.-W. Chao, M. Cakmak, and D. Fox, “Learning perceptual concepts by bootstrapping from human queries,”

2022

Closest in time.

F. F. Xu, U. Alon, G. Neubig, and V. J. Hellendoorn, “A systematic evaluation of large language models of code,” in

2022

Closest in time.

K. Zakka, A. Zeng, P. Florence, J. Tompson, J. Bohg, and D. Dwibedi, “Xirl: Cross-embodiment inverse reinforcement learning,” in

2022

Closest in time.

A. Ganapathi, P. Florence, J. Varley, K. Burns, K. Goldberg, and A. Zeng, “Implicit kinematic policies: Unifying joint and cartesian action spaces in end-to-end robot learning,”

Original

2022

Closest in time.