Minions: Accelerating Large Language Model Inference with Aggregated Speculative Execution

Understand

Large language models (LLM) have recently attracted surging interest due to their outstanding capabilities across various domains.

• However, enabling efficient LLM inference is challenging due to its autoregressive decoding that generates tokens only one at a time.
• Although research works apply pruning or quantization to speed up LLM inference, they typically require fine-tuning the LLM, incurring significant time and economic costs.
• Meanwhile, speculative decoding has been proposed to use small speculative models (SSMs) to accelerate the inference of LLM.