Layer-Order Inversion: Rethinking Latent Multi-Hop Reasoning in Large Language Models

Understand

Large language models (LLMs) perform well on multi-hop reasoning, yet how they internally compose multiple facts remains unclear.

• Recent work proposes \emph{hop-aligned circuit hypothesis}, suggesting that bridge entities are computed sequentially across layers before later-hop answers.
• Through systematic analyses on real-world multi-hop queries, we show that this hop-aligned assumption does not generalize: later-hop answer entities can become decodable earlier than bridge entities, a phenomenon we call \emph{layer-order inversion}, which strengthens with total hops.
• To explain this behavior, we propose a \emph{probabilistic recall-and-extract} framework that models multi-hop reasoning as broad probabilistic recall in shallow MLP layers followed by selective extraction in deeper attention layers.