Reasoning Elicitation in Language Models via Counterfactual Feedback

Understand

Despite the increasing effectiveness of language models, their reasoning capabilities remain underdeveloped.

• In particular, causal reasoning through counterfactual question answering is lacking.
• This work aims to bridge this gap.
• We first derive novel metrics that balance accuracy in factual and counterfactual questions, capturing a more complete view of the reasoning abilities of language models than traditional factual-only based metrics.