2023

Errors are Useful Prompts: Instruction Guided Task Programming with Verifier-Assisted Iterative Prompting

Skreta, Marta, Yoshikawa, Naruki, Arellano-Rubach, Sebastian et al.

Understand

Generating low-level robot task plans from high-level natural language instructions remains a challenging problem.

Although large language models have shown promising results in generating plans, the accuracy of the output remains unverified.
Furthermore, the lack of domain-specific language data poses a limitation on the applicability of these models.
In this paper, we propose CLAIRIFY, a novel approach that combines automatic iterative prompting with program verification to ensure programs written in data-scarce domain-specific language are syntactically valid and incorporate environment constraints.

Built on

J. J. Fortman and K. M. Stubbs, “Demonstrations with red cabbage indicator,” J. Chem. Educ. , vol. 69, no. 1, p. 66, 1992
1992
Earlier work this paper cites.
J. A. Baier, et al. , “A heuristic search approach to planning with temporally extended preferences,” Artif. Intell. , 2009
2009
Earlier work this paper cites.
L. P. Kaelbling and T. Lozano-Pérez, “Hierarchical task and motion planning in the now,” in IEEE Inter. Conf. Robot. Autom. IEEE, 2011, pp. 1470–1477
2011
Earlier work this paper cites.
D. Xu, et al. , “Neural task programming: Learning to generalize across hierarchical tasks,” in IEEE Int. Conf. Robot. Autom. , 2018
2018
Earlier work this paper cites.
J. Devlin, et al. , “BERT: Pre-training of deep bidirectional transformers for language understanding,” in North American Chapter of the Association for Computational Linguistics , 2019
2019
Earlier work this paper cites.
D.-A. Huang, et al. , “Neural Task Graphs: Generalizing to unseen tasks from a single video demonstration,” in IEEE Comput. Vis. Pattern Recognit. , 2019
2019
Earlier work this paper cites.
D. Xu, et al. , “Regression planning networks,” Adv. Neural. Inf. Process. Syst. , vol. 32, 2019
2019
Earlier work this paper cites.
B. Eysenbach, et al. , “Search on the replay buffer: Bridging planning and reinforcement learning,” Adv. Neural. Inf. Process. Syst. , vol. 32, 2019
2019
Earlier work this paper cites.
T. Brown, et al. , “Language models are few-shot learners,” Adv. Neural. Inf. Process. Syst. , vol. 33, pp. 1877–1901, 2020
2020
Earlier work this paper cites.
C. R. Garrett, et al. , “PDDLStream: Integrating symbolic planners and blackbox samplers via optimistic adaptive planning,” in Proc. ICAPS Conf. AAAI Press, 2020, pp. 440–448
2020
Earlier work this paper cites.
S. H. M. Mehr, et al. , “A universal system for digitization and automatic execution of the chemical synthesis literature,” Science , vol. 370, no. 6512, pp. 101–108, 2020
2020
Earlier work this paper cites.
A. C. Vaucher, et al. , “Automated extraction of chemical synthesis actions from experimental procedures,” Nat. Commun. , 2020
2020
Earlier work this paper cites.

Similar

H. Mehr, et al. , “Benchmarking results and the XDL XML schema.” 2020. [Online]. Available: https://zenodo.org/record/3955107
2020
Cited alongside, same era.
M. Chen, et al. , “Evaluating large language models trained on code,” arXiv:2107.03374 , 2021
Original
2021
Cited alongside, same era.
Y. Wang, et al. , “CodeT5: Identifier-aware unified pre-trained encoder-decoder models for code understanding and generation,” arXiv:2109.00859 , 2021
Original
2021
Cited alongside, same era.
Y. Gu, et al. , “Domain-specific language model pretraining for biomedical natural language processing,” ACM Trans. Comput. Healthcare , vol. 3, no. 1, pp. 1–23, 2021
2021
Cited alongside, same era.
S. Wang, et al. , “Want to reduce labeling cost? GPT-3 can help,” in Proc. EMNLP Conf. , 2021, pp. 4195–4205
2021
Cited alongside, same era.
N. Bannour, et al. , “Evaluating the carbon footprint of NLP methods: a survey and analysis of existing tools,” in SustaiNLP , Nov. 2021, pp. 11–21
2021
Cited alongside, same era.
S. Mishra, et al. , “Cross-task generalization via natural language crowdsourcing instructions,” arXiv:2104.08773 , 2021
Original
2021
Cited alongside, same era.
P. Sharma, et al. , “Skill induction and planning with latent language,” arXiv:2110.01517 , 2021
Original
2021
Cited alongside, same era.
S. Mirchandani, et al. , “ELLA: Exploration through learned language abstraction,” Adv. Neural. Inf. Process. Syst. , 2021
2021
Cited alongside, same era.
D. Shah, et al. , “Value function spaces: Skill-centric state abstractions for long-horizon reasoning,” arXiv:2111.03189 , 2021
Original
2021
Cited alongside, same era.
T. Perry, “LightTag: Text annotation platform,” in Proc. EMNLP Conf. , Nov. 2021, pp. 20–27
2021
Cited alongside, same era.
A. Chowdhery, et al. , “PaLM: Scaling language modeling with pathways,” arXiv:2204.02311 , 2022
Original
2022
Cited alongside, same era.

Then

Y. Li, et al. , “Competition-level code generation with alphacode,” Science , vol. 378, no. 6624, pp. 1092–1097, 2022
2022
Later among the works it cites.
J. Liang, et al. , “Code as policies: Language model programs for embodied control,” arXiv:2209.07753 , 2022
Original
2022
Later among the works it cites.
M. Ahn, et al. , “Do As I Can, Not As I Say: Grounding language in robotic affordances,” arXiv:2204.01691 , 2022
Original
2022
Later among the works it cites.
R. Liu, et al. , “Mind’s eye: Grounded language model reasoning through simulation,” arXiv:2210.05359 , 2022
Original
2022
Later among the works it cites.
M. Seifrid, et al. , “Autonomous chemical experiments: Challenges and perspectives on establishing a self-driving lab,” Acc. Chem. Res. , 2022
2022
Later among the works it cites.
W. Huang, et al. , “Inner monologue: Embodied reasoning through planning with language models,” arXiv:2207.05608 , 2022
Original
2022
Later among the works it cites.
H. Le, et al. , “CodeRL: Mastering code generation through pretrained models and deep reinforcement learning,” in Adv. Neural. Inf. Process. Syst. , 2022
2022
Later among the works it cites.
N. Yoshikawa, et al. , “An adaptive robotics framework for chemistry lab automation,” arXiv:2212.09672 , 2022
Original
2022
Later among the works it cites.
I. Singh, et al. , “Progprompt: Generating situated robot task plans using large language models,” in IEEE Int. Conf. Robot. Autom. , 2023
2023
Closest in time.
T. Schick, et al. , “Toolformer: Language models can teach themselves to use tools,” arXiv:2302.04761 , 2023
Original
2023
Closest in time.
A. Ni, et al. , “Lever: Learning to verify language-to-code generation with execution,” arXiv:2302.08468 , 2023
Original
2023
Closest in time.
B. Peng, et al. , “Check your facts and try again: Improving large language models with external knowledge and automated feedback,” arXiv:2302.12813 , 2023
Original
2023
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…