Built on

Codebleu: a method for automatic evaluation of code synthesis
Original
Shuo Ren, Daya Guo, Shuai Lu, Long Zhou, Shujie Liu, Duyu Tang, Neel Sundaresan, Ming Zhou, Ambrosio Blanco, and Shuai Ma. 2020 · 2009
Earlier work this paper cites.
Reclor: A reading comprehension dataset requiring logical reasoning
Weihao Yu, Zihang Jiang, Yanfei Dong, and Jiashi Feng. 2020 · 2020
Earlier work this paper cites.
Training verifiers to solve math word problems
Original
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. 2021 · 2021
Earlier work this paper cites.
proscript: Partially ordered scripts generation via pre-trained language models
Original
Keisuke Sakaguchi, Chandra Bhagavatula, Ronan Le Bras, Niket Tandon, Peter Clark, and Yejin Choi. 2021 · 2021
Earlier work this paper cites.
Folio: Natural language reasoning with first-order logic
Original
Simeng Han, Hailey Schoelkopf, Yilun Zhao, Zhenting Qi, Martin Riddell, Luke Benson, Lucy Sun, Ekaterina Zubova, Yujie Qiao, Matthew Burtell, David Peng, Jonathan Fan, Yixin Liu, Brian Wong, Malcolm Sailor, Ansong Ni, Linyong Nan, Jungo Kasai, Tao Yu, Rui Zhang, Shafiq Joty, Alexander R. Fabbri, Wojciech Kryscinski, Xi Victoria Lin, Caiming Xiong, and Dragomir Radev. 2022 · 2022
Earlier work this paper cites.
React: Synergizing reasoning and acting in language models
Original
Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. 2022 · 2022
Earlier work this paper cites.
Language Models can be Logical Solvers
Original
Jiazhan Feng, Ruochen Xu, Junheng Hao, Hiteshi Sharma, Yelong Shen, Dongyan Zhao, and Weizhu Chen. 2023 · 2023
Earlier work this paper cites.

Similar

PAL: Program-aided Language Models
Original
Luyu Gao, Aman Madaan, Shuyan Zhou, Uri Alon, Pengfei Liu, Yiming Yang, Jamie Callan, and Graham Neubig. 2023 · 2023
Cited alongside, same era.
Tora: A tool-integrated reasoning agent for mathematical problem solving
Original
Zhibin Gou, Zhihong Shao, Yeyun Gong, Yujiu Yang, Minlie Huang, Nan Duan, Weizhu Chen, et al. 2023 · 2023
Cited alongside, same era.
ToolkenGPT: Augmenting Frozen Language Models with Massive Tools via Tool Embeddings
Original
Shibo Hao, Tianyang Liu, Zhen Wang, and Zhiting Hu. 2023 · 2023
Cited alongside, same era.
Program-Aided Reasoners (better) Know What They Know
Original
Anubha Kabra, Sanketh Rangreji, Yash Mathur, Aman Madaan, Emmy Liu, and Graham Neubig. 2023 · 2023
Cited alongside, same era.

Then

Cognitive architectures for language agents
Original
Theodore R. Sumers, Shunyu Yao, Karthik Narasimhan, and Thomas L. Griffiths. 2023 · 2023
Later among the works it cites.
Harnessing the power of large language models for natural language to first-order logic translation
Original
Yuan Yang, Siheng Xiong, Ali Payani, Ehsan Shareghi, and Faramarz Fekri. 2023 · 2023
Later among the works it cites.
SATLM: Satisfiability-Aided Language Models Using Declarative Prompting
Xi Ye, Qiaochu Chen, Isil Dillig, and Greg Durrett. 2023 · 2023
Later among the works it cites.
Isabelle overview
Isabelle. 2024 · 2024
Closest in time.
Lean manual
FRO Lean. 2024 · 2024
Closest in time.
Advancing LLM Reasoning Generalists with Preference Trees

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…

2024

Can LLMs Reason in the Wild with Programs?

Yang, Yuan, Xiong, Siheng, Payani, Ali et al.

Understand

Large Language Models (LLMs) have shown superior capability to solve reasoning problems with programs.

While being a promising direction, most of such frameworks are trained and evaluated in settings with a prior knowledge of task requirements.
However, as LLMs become more capable, it is necessary to assess their reasoning abilities in more realistic scenarios where many real-world problems are open-ended with ambiguous scope, and often require multiple formalisms to solve.
To investigate this, we introduce the task of reasoning in the wild, where an LLM is tasked to solve a reasoning problem of unknown type by identifying the subproblems and their corresponding formalisms, and writing a program to solve each subproblem, guided by a tactic.

Built on

Codebleu: a method for automatic evaluation of code synthesis
Original
Shuo Ren, Daya Guo, Shuai Lu, Long Zhou, Shujie Liu, Duyu Tang, Neel Sundaresan, Ming Zhou, Ambrosio Blanco, and Shuai Ma. 2020 · 2009
Earlier work this paper cites.
Reclor: A reading comprehension dataset requiring logical reasoning
Weihao Yu, Zihang Jiang, Yanfei Dong, and Jiashi Feng. 2020 · 2020
Earlier work this paper cites.
Training verifiers to solve math word problems
Original
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. 2021 · 2021
Earlier work this paper cites.
proscript: Partially ordered scripts generation via pre-trained language models
Original
Keisuke Sakaguchi, Chandra Bhagavatula, Ronan Le Bras, Niket Tandon, Peter Clark, and Yejin Choi. 2021 · 2021
Earlier work this paper cites.
Folio: Natural language reasoning with first-order logic
Original
Simeng Han, Hailey Schoelkopf, Yilun Zhao, Zhenting Qi, Martin Riddell, Luke Benson, Lucy Sun, Ekaterina Zubova, Yujie Qiao, Matthew Burtell, David Peng, Jonathan Fan, Yixin Liu, Brian Wong, Malcolm Sailor, Ansong Ni, Linyong Nan, Jungo Kasai, Tao Yu, Rui Zhang, Shafiq Joty, Alexander R. Fabbri, Wojciech Kryscinski, Xi Victoria Lin, Caiming Xiong, and Dragomir Radev. 2022 · 2022
Earlier work this paper cites.
React: Synergizing reasoning and acting in language models
Original
Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. 2022 · 2022
Earlier work this paper cites.
Language Models can be Logical Solvers
Original
Jiazhan Feng, Ruochen Xu, Junheng Hao, Hiteshi Sharma, Yelong Shen, Dongyan Zhao, and Weizhu Chen. 2023 · 2023
Earlier work this paper cites.

Similar

PAL: Program-aided Language Models
Original
Luyu Gao, Aman Madaan, Shuyan Zhou, Uri Alon, Pengfei Liu, Yiming Yang, Jamie Callan, and Graham Neubig. 2023 · 2023
Cited alongside, same era.
Tora: A tool-integrated reasoning agent for mathematical problem solving
Original
Zhibin Gou, Zhihong Shao, Yeyun Gong, Yujiu Yang, Minlie Huang, Nan Duan, Weizhu Chen, et al. 2023 · 2023
Cited alongside, same era.
ToolkenGPT: Augmenting Frozen Language Models with Massive Tools via Tool Embeddings
Original
Shibo Hao, Tianyang Liu, Zhen Wang, and Zhiting Hu. 2023 · 2023
Cited alongside, same era.
Program-Aided Reasoners (better) Know What They Know
Original
Anubha Kabra, Sanketh Rangreji, Yash Mathur, Aman Madaan, Emmy Liu, and Graham Neubig. 2023 · 2023
Cited alongside, same era.

Then

Cognitive architectures for language agents
Original
Theodore R. Sumers, Shunyu Yao, Karthik Narasimhan, and Thomas L. Griffiths. 2023 · 2023
Later among the works it cites.
Harnessing the power of large language models for natural language to first-order logic translation
Original
Yuan Yang, Siheng Xiong, Ali Payani, Ehsan Shareghi, and Faramarz Fekri. 2023 · 2023
Later among the works it cites.
SATLM: Satisfiability-Aided Language Models Using Declarative Prompting
Xi Ye, Qiaochu Chen, Isil Dillig, and Greg Durrett. 2023 · 2023
Later among the works it cites.
Isabelle overview
Isabelle. 2024 · 2024
Closest in time.
Lean manual
FRO Lean. 2024 · 2024
Closest in time.
Advancing LLM Reasoning Generalists with Preference Trees

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

LINC: A Neurosymbolic Approach for Logical Reasoning by Combining Language Models with First-Order Logic Provers

Original

Theo X. Olausson, Alex Gu, Benjamin Lipkin, Cedegao E. Zhang, Armando Solar-Lezama, Joshua B. Tenenbaum, and Roger Levy. 2023 · 2023

Cited alongside, same era.

Logic-LM: Empowering Large Language Models with Symbolic Solvers for Faithful Logical Reasoning

Original

Liangming Pan, Alon Albalak, Xinyi Wang, and William Yang Wang. 2023 · 2023

Cited alongside, same era.

Toolformer: Language Models Can Teach Themselves to Use Tools

Original

Timo Schick, Jane Dwivedi-Yu, Roberto Dessì, Roberta Raileanu, Maria Lomeli, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. 2023 · 2023

Cited alongside, same era.

Fireact: Toward language agent fine-tuning

Original

Baian Chen, Chang Shu, Ehsan Shareghi, Nigel Collier, Karthik Narasimhan, and Shunyu Yao. 2023a

Cited in the paper.

Program of Thoughts Prompting: Disentangling Computation from Reasoning for Numerical Reasoning Tasks

Original

Wenhu Chen, Xueguang Ma, Xinyi Wang, and William W. Cohen. 2023b

Cited in the paper.

Original

Lifan Yuan, Ganqu Cui, Hanbin Wang, Ning Ding, Xingyao Wang, Jia Deng, Boji Shan, Huimin Chen, Ruobing Xie, Yankai Lin, Zhenghao Liu, Bowen Zhou, Hao Peng, Zhiyuan Liu, and Maosong Sun. 2024 · 2024

Closest in time.