Polar Sparsity: High Throughput Batched LLM Inferencing with Scalable Contextual Sparsity

Understand

Accelerating large language model (LLM) inference is critical for real-world deployments requiring high throughput and low latency.

• Contextual sparsity, where each token dynamically activates only a small subset of the model parameters, shows promise but does not scale to large batch sizes due to union of active neurons quickly approaching dense computation.
• We introduce Polar Sparsity, highlighting a key shift in sparsity importance from MLP to Attention layers as we scale batch size and sequence length.
• While MLP layers become more compute-efficient under batching, their sparsity vanishes.