LeanQuant: Accurate and Scalable Large Language Model Quantization with Loss-error-aware Grid

Understand

Large language models (LLMs) have shown immense potential across various domains, but their high memory requirements and inference costs remain critical challenges for deployment.

• Post-training quantization (PTQ) has emerged as a promising technique to reduce memory requirements and decoding latency.
• However, recent accurate quantization methods often depend on specialized computations or custom data formats to achieve better model quality, which limits their compatibility with popular frameworks, as they require dedicated inference kernels tailored to specific hardware and software platforms, hindering wider adoption.
• Furthermore, many competitive methods have high resource requirements and computational overhead for quantizing models, making it challenging to scale them to hundreds of billions of parameters.