Entropy-Lens: Uncovering Decision Strategies in LLMs

Understand

In large language models (LLMs), each block operates on the residual stream to map input token sequences to output token distributions.

• However, most of the interpretability literature focuses on internal latent representations, leaving token-space dynamics underexplored.
• The high dimensionality and categoricity of token distributions hinder their analysis, as standard statistical descriptors are not suitable.
• We show that the entropy of logit-lens predictions overcomes these issues.