Efficient LLM Inference on CPUs

Understand

Large language models (LLMs) have demonstrated remarkable performance and tremendous potential across a wide range of tasks.

• However, deploying these models has been challenging due to the astronomical amount of model parameters, which requires a demand for large memory capacity and high memory bandwidth.
• In this paper, we propose an effective approach that can make the deployment of LLMs more efficiently.
• We support an automatic INT4 weight-only quantization flow and design a special LLM runtime with highly-optimized kernels to accelerate the LLM inference on CPUs.