LogicBench: Towards Systematic Evaluation of Logical Reasoning Ability of Large Language Models

Understand

Recently developed large language models (LLMs) have been shown to perform remarkably well on a wide range of language understanding tasks.

• But, can they really "reason" over the natural language? This question has been receiving significant research attention and many reasoning skills such as commonsense, numerical, and qualitative have been studied.
• However, the crucial skill pertaining to 'logical reasoning' has remained underexplored.
• Existing work investigating this reasoning ability of LLMs has focused only on a couple of inference rules (such as modus ponens and modus tollens) of propositional and first-order logic.