QQQ: Quality Quattuor-Bit Quantization for Large Language Models

Understand

Quantization is a proven effective method for compressing large language models.

• Although popular techniques like W8A8 and W4A16 effectively maintain model performance, they often fail to concurrently speed up the prefill and decoding stages of inference.
• W4A8 is a promising strategy to accelerate both of them while usually leads to a significant performance degradation.
• To address these issues, we present QQQ, a Quality Quattuor-bit Quantization method with 4-bit weights and 8-bit activations.