EasySpec: Layer-Parallel Speculative Decoding for Efficient Multi-GPU Utilization

Understand

Speculative decoding is an effective and lossless method for Large Language Model (LLM) inference acceleration.

• It employs a smaller model to generate a draft token sequence, which is then verified by the original base model.
• In multi-GPU systems, inference latency can be further reduced through tensor parallelism (TP), while the optimal TP size of the draft model is typically smaller than that of the base model, leading to GPU idling during the drafting stage.
• We observe that such inefficiency stems from the sequential execution of layers, which is seemingly natural but actually unnecessary.