Disentangling Neuron Representations with Concept Vectors

Understand

Mechanistic interpretability aims to understand how models store representations by breaking down neural networks into interpretable units.

• However, the occurrence of polysemantic neurons, or neurons that respond to multiple unrelated features, makes interpreting individual neurons challenging.
• This has led to the search for meaningful vectors, known as concept vectors, in activation space instead of individual neurons.
• The main contribution of this paper is a method to disentangle polysemantic neurons into concept vectors encapsulating distinct features.