A Closer Look at Logical Reasoning with LLMs: The Choice of Tool Matters

Understand

The emergence of Large Language Models (LLMs) has demonstrated promising progress in solving logical reasoning tasks effectively.

• Several recent approaches have proposed to change the role of the LLM from the reasoner into a translator between natural language statements and symbolic representations which are then sent to external symbolic solvers to resolve.
• This paradigm has established the current state-of-the-art result in logical reasoning (i.e., deductive reasoning).
• However, it remains unclear whether the variance in performance of these approaches stems from the methodologies employed or the specific symbolic solvers utilized.