Spectral Filters, Dark Signals, and Attention Sinks

Understand

Projecting intermediate representations onto the vocabulary is an increasingly popular interpretation tool for transformer-based LLMs, also known as the logit lens.

• We propose a quantitative extension to this approach and define spectral filters on intermediate representations based on partitioning the singular vectors of the vocabulary embedding and unembedding matrices into bands.
• We find that the signals exchanged in the tail end of the spectrum are responsible for attention sinking (Xiao et al.
• 2023), of which we provide an explanation.