Traversal Verification for Speculative Tree Decoding

Understand

Speculative decoding is a promising approach for accelerating large language models.

• The primary idea is to use a lightweight draft model to speculate the output of the target model for multiple subsequent timesteps, and then verify them in parallel to determine whether the drafted tokens should be accepted or rejected.
• To enhance acceptance rates, existing frameworks typically construct token trees containing multiple candidates in each timestep.
• However, their reliance on token-level verification mechanisms introduces two critical limitations: First, the probability distribution of a sequence differs from that of individual tokens, leading to suboptimal acceptance length.