Built on

C. E. Shannon, “A mathematical theory of communication,” The Bell system technical journal
1948
Earlier work this paper cites.
D. A. Huffman, “A method for the construction of minimum-redundancy codes,” Proceedings of the IRE
1952
Earlier work this paper cites.
J. Ziv and A. Lempel, “A universal algorithm for sequential data compression,” IEEE Transactions on information theory
1977
Earlier work this paper cites.
P. Deutsch, “Deflate compressed data format specification version 1.3,” tech. rep., 1996
1996
Earlier work this paper cites.
P. Deutsch, “GZIP file format specification version 4.3,” RFC
1996
Earlier work this paper cites.
J. Duda, “Asymmetric numeral systems: entropy coding combining speed of huffman coding with compression rate of arithmetic coding,” arXiv preprint arXiv:1311.2540
Original
2013
Earlier work this paper cites.
S. Han, H. Mao, and W. J. Dally, “Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding,” arXiv preprint arXiv:1510.00149
Original
2015
Earlier work this paper cites.
Y. Collet, “Zstd github repository from facebook.,” 2016
2016
Earlier work this paper cites.
P. Jain, A. Jain, A. Nrusimha, A. Gholami, P. Abbeel, J. Gonzalez, K. Keutzer, and I. Stoica, “Checkmate: Breaking the memory wall with optimal tensor rematerialization,” Proceedings of Machine Learning and Systems
2020
Earlier work this paper cites.
J. Ren, S. Rajbhandari, R. Y. Aminabadi, O. Ruwase, S. Yang, M. Zhang, D. Li, and Y. He, “ { \{ ZeRO-Offload } \} : Democratizing { \{ Billion-Scale } \} model training,” in 2021 USENIX Annual Technical Conference (USENIX ATC 21)
2021
Earlier work this paper cites.
T. Dao, D. Fu, S. Ermon, A. Rudra, and C. Ré, “Flashattention: Fast and memory-efficient exact attention with io-awareness,” Advances in Neural Information Processing Systems
2022
Earlier work this paper cites.

Similar

E. Frantar, S. Ashkboos, T. Hoefler, and D. Alistarh, “Gptq: Accurate post-training quantization for generative pre-trained transformers,” arXiv preprint arXiv:2210.17323
Original
2022
Cited alongside, same era.
T. Dettmers, M. Lewis, Y. Belkada, and L. Zettlemoyer, “Llm. int8 (): 8-bit matrix multiplication for transformers at scale,” arXiv preprint arXiv:2208.07339
Original
2022
Cited alongside, same era.
S. Smith, M. Patwary, B. Norick, P. LeGresley, S. Rajbhandari, J. Casper, Z. Liu, S. Prabhumoye, G. Zerveas, V. Korthikanti, et al
Original
2022
Cited alongside, same era.
Z. Yao, R. Yazdani Aminabadi, M. Zhang, X. Wu, C. Li, and Y. He, “Zeroquant: Efficient and affordable post-training quantization for large-scale transformers,” Advances in Neural Information Processing Systems
2022
Cited alongside, same era.

Then

J. Kaddour, J. Harris, M. Mozes, H. Bradley, R. Raileanu, and R. McHardy, “Challenges and applications of large language models,” arXiv preprint arXiv:2307.10169
Original
2023
Later among the works it cites.
G. Xiao, J. Lin, M. Seznec, H. Wu, J. Demouth, and S. Han, “Smoothquant: Accurate and efficient post-training quantization for large language models,” in International Conference on Machine Learning
2023
Later among the works it cites.
S.-y. Liu, Z. Liu, X. Huang, P. Dong, and K.-T. Cheng, “Llm-fp4: 4-bit floating-point quantized transformers,” arXiv preprint arXiv:2310.16836
Original
2023
Later among the works it cites.
V. A. Korthikanti, J. Casper, S. Lym, L. McAfee, M. Andersch, M. Shoeybi, and B. Catanzaro, “Reducing activation recomputation in large transformer models,” Proceedings of Machine Learning and Systems
2023
Later among the works it cites.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…

2024

On the Compressibility of Quantized Large Language Models

Mao, Yu, Wang, Weilan, Du, Hongchao et al.

Understand

Deploying Large Language Models (LLMs) on edge or mobile devices offers significant benefits, such as enhanced data privacy and real-time processing capabilities.

However, it also faces critical challenges due to the substantial memory requirement of LLMs.
Quantization is an effective way of reducing the model size while maintaining good performance.
However, even after quantization, LLMs may still be too big to fit entirely into the limited memory of edge or mobile devices and have to be partially loaded from the storage to complete the inference.

Built on

C. E. Shannon, “A mathematical theory of communication,” The Bell system technical journal
1948
Earlier work this paper cites.
D. A. Huffman, “A method for the construction of minimum-redundancy codes,” Proceedings of the IRE
1952
Earlier work this paper cites.
J. Ziv and A. Lempel, “A universal algorithm for sequential data compression,” IEEE Transactions on information theory
1977
Earlier work this paper cites.
P. Deutsch, “Deflate compressed data format specification version 1.3,” tech. rep., 1996
1996
Earlier work this paper cites.
P. Deutsch, “GZIP file format specification version 4.3,” RFC
1996
Earlier work this paper cites.
J. Duda, “Asymmetric numeral systems: entropy coding combining speed of huffman coding with compression rate of arithmetic coding,” arXiv preprint arXiv:1311.2540
Original
2013
Earlier work this paper cites.
S. Han, H. Mao, and W. J. Dally, “Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding,” arXiv preprint arXiv:1510.00149
Original
2015
Earlier work this paper cites.
Y. Collet, “Zstd github repository from facebook.,” 2016
2016
Earlier work this paper cites.
P. Jain, A. Jain, A. Nrusimha, A. Gholami, P. Abbeel, J. Gonzalez, K. Keutzer, and I. Stoica, “Checkmate: Breaking the memory wall with optimal tensor rematerialization,” Proceedings of Machine Learning and Systems
2020
Earlier work this paper cites.
J. Ren, S. Rajbhandari, R. Y. Aminabadi, O. Ruwase, S. Yang, M. Zhang, D. Li, and Y. He, “ { \{ ZeRO-Offload } \} : Democratizing { \{ Billion-Scale } \} model training,” in 2021 USENIX Annual Technical Conference (USENIX ATC 21)
2021
Earlier work this paper cites.
T. Dao, D. Fu, S. Ermon, A. Rudra, and C. Ré, “Flashattention: Fast and memory-efficient exact attention with io-awareness,” Advances in Neural Information Processing Systems
2022
Earlier work this paper cites.

Similar

E. Frantar, S. Ashkboos, T. Hoefler, and D. Alistarh, “Gptq: Accurate post-training quantization for generative pre-trained transformers,” arXiv preprint arXiv:2210.17323
Original
2022
Cited alongside, same era.
T. Dettmers, M. Lewis, Y. Belkada, and L. Zettlemoyer, “Llm. int8 (): 8-bit matrix multiplication for transformers at scale,” arXiv preprint arXiv:2208.07339
Original
2022
Cited alongside, same era.
S. Smith, M. Patwary, B. Norick, P. LeGresley, S. Rajbhandari, J. Casper, Z. Liu, S. Prabhumoye, G. Zerveas, V. Korthikanti, et al
Original
2022
Cited alongside, same era.
Z. Yao, R. Yazdani Aminabadi, M. Zhang, X. Wu, C. Li, and Y. He, “Zeroquant: Efficient and affordable post-training quantization for large-scale transformers,” Advances in Neural Information Processing Systems
2022
Cited alongside, same era.

Then

J. Kaddour, J. Harris, M. Mozes, H. Bradley, R. Raileanu, and R. McHardy, “Challenges and applications of large language models,” arXiv preprint arXiv:2307.10169
Original
2023
Later among the works it cites.
G. Xiao, J. Lin, M. Seznec, H. Wu, J. Demouth, and S. Han, “Smoothquant: Accurate and efficient post-training quantization for large language models,” in International Conference on Machine Learning
2023
Later among the works it cites.
S.-y. Liu, Z. Liu, X. Huang, P. Dong, and K.-T. Cheng, “Llm-fp4: 4-bit floating-point quantized transformers,” arXiv preprint arXiv:2310.16836
Original
2023
Later among the works it cites.
V. A. Korthikanti, J. Casper, S. Lym, L. McAfee, M. Andersch, M. Shoeybi, and B. Catanzaro, “Reducing activation recomputation in large transformer models,” Proceedings of Machine Learning and Systems
2023
Later among the works it cites.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Y. Mao et al

2022

Cited alongside, same era.

Y. Mao et al

2022

Cited alongside, same era.

S. Zhang, S. Roller, N. Goyal, M. Artetxe, M. Chen, S. Chen, C. Dewan, M. Diab, X. Li, X. V. Lin, et al

Original

2022

Cited alongside, same era.

Y. Sheng, L. Zheng, B. Yuan, Z. Li, M. Ryabinin, B. Chen, P. Liang, C. Ré, I. Stoica, and C. Zhang, “Flexgen: High-throughput generative inference of large language models with a single gpu,” in International Conference on Machine Learning

2023

Cited alongside, same era.

K. Alizadeh, I. Mirzadeh, D. Belenko, K. Khatamifard, M. Cho, C. C. Del Mundo, M. Rastegari, and M. Farajtabar, “Llm in a flash: Efficient large language model inference with limited memory,” arXiv preprint arXiv:2312.11514

Original

2023

Cited alongside, same era.

L. Zhang, X. Liu, Z. Li, X. Pan, P. Dong, R. Fan, R. Guo, X. Wang, Q. Luo, S. Shi, et al

Original

2023

Cited alongside, same era.

J. Lin, J. Tang, H. Tang, S. Yang, X. Dang, and S. Han, “Awq: Activation-aware weight quantization for llm compression and acceleration,” arXiv preprint arXiv:2306.00978

Original

2023