Multiple Choice Questions: Reasoning Makes Large Language Models (LLMs) More Self-Confident, Especially When They are Wrong

Understand

Multiple Choice Question (MCQ) tests are among the most used methods for evaluating large language models (LLMs).

• Besides checking the correctness of the selected answer, evaluations often consider the model's confidence through the probability assigned to its response.
• In this work, we investigate how LLM confidence is influenced by the answering approach when the model answers directly or reasons before responding.
• Experiments on a general knowledge benchmark, covering 57 subjects and seven LLMs, show that models are systematically more confident when providing reasoning before answering, and that this confidence increase is larger when the selected answer is incorrect than when it is correct.