Speculative Decoding via Early-exiting for Faster LLM Inference with Thompson Sampling Control Mechanism

Understand

The recent advancements in large language models (LLMs) have been extraordinary, yet the escalating inference costs associated with them present challenges in real-world applications.

• To address these challenges, we propose a novel approach called Early-exiting Speculative Decoding (EESD) with lossless acceleration.
• Specifically, EESD utilizes a segment of the LLM to generate draft tokens, incorporating Early-exiting structures after the first N layers.
• To enhance the quality of draft tokens, a self-distillation method is integrated.