eDKM: An Efficient and Accurate Train-time Weight Clustering for Large Language Models

Understand

Since Large Language Models or LLMs have demonstrated high-quality performance on many complex language tasks, there is a great interest in bringing these LLMs to mobile devices for faster responses and better privacy protection.

• However, the size of LLMs (i.e., billions of parameters) requires highly effective compression to fit into storage-limited devices.
• Among many compression techniques, weight-clustering, a form of non-linear quantization, is one of the leading candidates for LLM compression, and supported by modern smartphones.
• Yet, its training overhead is prohibitively significant for LLM fine-tuning.