HiSpec: Hierarchical Speculative Decoding for LLMs

Understand

Speculative decoding accelerates LLM inference by using a smaller draft model to speculate tokens that a larger target model verifies.

• Verification is often the bottleneck (e.g.
• verification is $4\times$ slower than token generation when a 3B model speculates for a 70B target model), but most prior works focus only on accelerating drafting.
• $\textit{``Intermediate"}$ verification reduces verification time by discarding inaccurate draft tokens early, but existing methods incur substantial training overheads in incorporating the intermediate verifier, increase the memory footprint to orchestrate the intermediate verification step, and compromise accuracy by relying on approximate heuristics.