FlashDecoding++: Faster Large Language Model Inference on GPUs

Understand

As the Large Language Model (LLM) becomes increasingly important in various domains.

• However, the following challenges still remain unsolved in accelerating LLM inference: (1) Synchronized partial softmax update.
• The softmax operation requires a synchronized update operation among each partial softmax result, leading to ~20% overheads for the attention computation in LLMs.
• (2) Under-utilized computation of flat GEMM.