Hardware-Aware Parallel Prompt Decoding for Memory-Efficient Acceleration of LLM Inference

Understand

The auto-regressive decoding of Large Language Models (LLMs) results in significant overheads in their hardware performance.

• While recent research has investigated various speculative decoding techniques for multi-token generation, these efforts have primarily focused on improving processing speed such as throughput.
• Crucially, they often neglect other metrics essential for real-life deployments, such as memory consumption and training cost.
• To overcome these limitations, we propose a novel parallel prompt decoding that requires only $0.0002$% trainable parameters, enabling efficient training on a single A100-40GB GPU in just 16 hours.