CodeMind: Evaluating Large Language Models for Code Reasoning

Understand

Large Language Models (LLMs) have been widely used to automate programming tasks.

• Their capabilities have been evaluated by assessing the quality of generated code through tests or proofs.
• The extent to which they can reason about code is a critical question revealing important insights about their true capabilities.
• This paper introduces CodeMind, a framework designed to gauge the code reasoning abilities of LLMs through the following explicit and implicit code reasoning tasks: Independent Execution Reasoning (IER), Specification Reasoning (SR) and Dynamic Semantics Reasoning (DSR).