2025

GRIFFIN: Effective Token Alignment for Faster Speculative Decoding

Hu, Shijing, Li, Jingyang, Xie, Xingyu et al.

Understand

Speculative decoding accelerates inference in large language models (LLMs) by generating multiple draft tokens simultaneously.

  • However, existing methods often struggle with token misalignment between the training and decoding phases, limiting their performance.
  • To address this, we propose GRIFFIN, a novel framework that incorporates a token-alignable training strategy and a token-alignable draft model to mitigate misalignment.
  • The training strategy employs a loss masking mechanism to exclude highly misaligned tokens during training, preventing them from negatively impacting the draft model's optimization.

Reading the bibliography…