Rigorously Assessing Natural Language Explanations of Neurons

Understand

Natural language is an appealing medium for explaining how large language models process and store information, but evaluating the faithfulness of such explanations is challenging.

• To help address this, we develop two modes of evaluation for natural language explanations that claim individual neurons represent a concept in a text input.
• In the observational mode, we evaluate claims that a neuron $a$ activates on all and only input strings that refer to a concept picked out by the proposed explanation $E$.
• In the intervention mode, we construe $E$ as a claim that the neuron $a$ is a causal mediator of the concept denoted by $E$.