When Attention Collapses: How Degenerate Layers in LLMs Enable Smaller, Stronger Models

Understand

Large Language Models (LLMs) are known for their performance, but we uncover a significant structural inefficiency: a phenomenon we term attention collapse.

• In many pre-trained decoder-style LLMs, the attention matrices in deeper layers degenerate, collapsing to near rank-one structures.
• These underutilized layers, which we call lazy layers, are redundant and impair model efficiency.
• To address this, we introduce Inheritune, a simple yet powerful training recipe designed to build smaller, stronger language models.