Fetching the paper…

Reading the bibliography…

2024

Accurate LoRA-Finetuning Quantization of LLMs via Information Retention

Qin, Haotong, Ma, Xudong, Zheng, Xingyu et al.

Original
PDF
Post on X

Understand

The LoRA-finetuning quantization of LLMs has been extensively studied to obtain accurate yet compact LLMs for deployment on resource-constrained hardware.

However, existing methods cause the quantized LLM to severely degrade and even fail to benefit from the finetuning of LoRA.
This paper proposes a novel IR-QLoRA for pushing quantized LLMs with LoRA to be highly accurate through information retention.
The proposed IR-QLoRA mainly relies on two technologies derived from the perspective of unified information: (1) statistics-based Information Calibration Quantization allows the quantized parameters of LLM to retain original information accurately; (2) finetuning-based Information Elastic Connection makes LoRA utilizes elastic representation transformation with diverse information.

Accurate LoRA-Finetuning Quantization of LLMs via Information Retention · Around

Built on

Think you have solved question answering? try arc, the ai2 reasoning challenge
Original
Clark, P., Cowhey, I., Etzioni, O., Khot, T., Sabharwal, A., Schoenick, C., and Tafjord, O · 2018
Earlier work this paper cites.
Bert: Pre-training of deep bidirectional transformers for language understanding
Original
Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K · 2018
Earlier work this paper cites.
Can a suit of armor conduct electricity? a new dataset for open book question answering
Mihaylov, T., Clark, P., Khot, T., and Sabharwal, A · 2018
Earlier work this paper cites.
Boolq: Exploring the surprising difficulty of natural yes/no questions
Original
Clark, C., Lee, K., Chang, M.-W., Kwiatkowski, T., Collins, M., and Toutanova, K · 2019
Earlier work this paper cites.
Hellaswag: Can a machine really finish your sentence?
Zellers, R., Holtzman, A., Bisk, Y., Farhadi, A., and Choi, Y · 2019
Earlier work this paper cites.
Piqa: Reasoning about physical commonsense in natural language
Bisk, Y., Zellers, R., Gao, J., Choi, Y., et al · 2020
Earlier work this paper cites.
Language models are few-shot learners
Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al · 2020
Earlier work this paper cites.
Measuring massive multitask language understanding
Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D., and Steinhardt, J · 2020
Earlier work this paper cites.
Uniq: Uniform noise injection for non-uniform quantization of neural networks
Baskin, C., Liss, N., Schwartz, E., Zheltonozhskii, E., Giryes, R., Bronstein, A. M., and Mendelson, A · 2021
Earlier work this paper cites.
8-bit optimizers via block-wise quantization
Original
Dettmers, T., Lewis, M., Shleifer, S., and Zettlemoyer, L · 2021
Earlier work this paper cites.
Compressing large-scale transformer-based models: A case study on bert
Ganesh, P., Chen, Y., Lou, X., Khan, M. A., Yang, Y., Sajjad, H., Nakov, P., Chen, D., and Winslett, M · 2021
Earlier work this paper cites.
Measuring massive multitask language understanding
Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D., and Steinhardt, J · 2021
Earlier work this paper cites.
Lora: Low-rank adaptation of large language models
Original
Hu, E. J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., and Chen, W · 2021
Earlier work this paper cites.
Winogrande: An adversarial winograd schema challenge at scale
Sakaguchi, K., Bras, R. L., Bhagavatula, C., and Choi, Y · 2021
Earlier work this paper cites.

Similar

Llm. int8 (): 8-bit matrix multiplication for transformers at scale
Original
Dettmers, T., Lewis, M., Belkada, Y., and Zettlemoyer, L · 2022
Cited alongside, same era.
Gptq: Accurate post-training quantization for generative pre-trained transformers
Original
Frantar, E., Ashkboos, S., Hoefler, T., and Alistarh, D · 2022
Cited alongside, same era.
Towards reasoning in large language models: A survey
Original
Huang, J. and Chang, K. C.-C · 2022
Cited alongside, same era.
In-context learning distillation: Transferring few-shot learning ability of pre-trained language models
Original
Huang, Y., Chen, Y., Yu, Z., and McKeown, K · 2022
Cited alongside, same era.
Explanations from large language models make small reasoners better

Then

Lion: Adversarial distillation of closed-source large language model
Original
Jiang, Y., Chan, C., Chen, M., and Wang, W · 2023
Later among the works it cites.
Memory-efficient fine-tuning of compressed large language models via sub-4-bit integer quantization
Original
Kim, J., Lee, J. H., Kim, S., Park, J., Yoo, K. M., Kwon, S. J., and Lee, D · 2023
Later among the works it cites.
Owq: Lessons learned from activation outliers for weight quantization in large language models
Original
Lee, C., Jin, J., Kim, T., Kim, H., and Park, E · 2023
Later among the works it cites.
Loftq: Lora-fine-tuning-aware quantization for large language models
Original
Li, Y., Yu, Y., Liang, C., He, P., Karampatziakis, N., Chen, W., and Zhao, T · 2023
Later among the works it cites.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Original

Li, S., Chen, J., Shen, Y., Chen, Z., Zhang, X., Li, Z., Wang, H., Qian, J., Peng, B., Mao, Y., et al · 2022

Cited alongside, same era.

Self-instruct: Aligning language model with self generated instructions

Wang, Y., Kordi, Y., Mishra, S., Liu, A., Smith, N. A., Khashabi, D., and Hajishirzi, H · 2022

Cited alongside, same era.

Gkd: Generalized knowledge distillation for auto-regressive sequence models

Agarwal, R., Vieillard, N., Stanczyk, P., Ramos, S., Geist, M., and Bachem, O · 2023

Cited alongside, same era.

A survey on evaluation of large language models

Chang, Y., Wang, X., Wang, J., Wu, Y., Zhu, K., Chen, H., Yang, L., Yi, X., Wang, C., Wang, Y., et al · 2023

Cited alongside, same era.

A survey of techniques for optimizing transformer inference

Chitty-Venkata, K. T., Mittal, S., Emani, M., Vishwanath, V., and Somani, A. K · 2023

Cited alongside, same era.

Palm: Scaling language modeling with pathways

Chowdhery, A., Narang, S., Devlin, J., Bosma, M., Mishra, G., Roberts, A., Barham, P., Chung, H. W., Sutton, C., Gehrmann, S., et al · 2023

Cited alongside, same era.

Qlora: Efficient finetuning of quantized llms

Dettmers, T., Pagnoni, A., Holtzman, A., and Zettlemoyer, L · 2023

Cited alongside, same era.

Sparsegpt: Massive language models can be accurately pruned in one-shot

Frantar, E. and Alistarh, D · 2023

Cited alongside, same era.

Awq: Activation-aware weight quantization for llm compression and acceleration

Lin, J., Tang, J., Tang, H., Yang, S., Dang, X., and Han, S · 2023

Later among the works it cites.

The flan collection: Designing data and methods for effective instruction tuning

Longpre, S., Hou, L., Vu, T., Webson, A., Chung, H. W., Tay, Y., Zhou, D., Le, Q. V., Zoph, B., Wei, J., et al · 2023

Later among the works it cites.

Distribution-sensitive information retention for accurate binary neural network

Qin, H., Zhang, X., Gong, R., Ding, Y., Xu, Y., and Liu, X · 2023

Later among the works it cites.

Omniquant: Omnidirectionally calibrated quantization for large language models

Shao, W., Chen, M., Zhang, Z., Xu, P., Zhao, L., Li, Z., Zhang, K., Gao, P., Qiao, Y., and Luo, P · 2023

Later among the works it cites.

A simple and effective pruning approach for large language models

Sun, M., Liu, Z., Bair, A., and Kolter, J. Z · 2023

Later among the works it cites.

Stanford alpaca: an instruction-following llama model (2023)

Taori, R., Gulrajani, I., Zhang, T., Dubois, Y., Li, X., Guestrin, C., Liang, P., and Hashimoto, T. B · 2023

Later among the works it cites.

Smoothquant: Accurate and efficient post-training quantization for large language models

Xiao, G., Lin, J., Seznec, M., Wu, H., Demouth, J., and Han, S · 2023

Later among the works it cites.

Pruning meets low-rank parameter-efficient fine-tuning

Zhang, M., Shen, C., Yang, Z., Ou, L., Yu, X., Zhuang, B., et al · 2023

Later among the works it cites.

A survey of large language models

Zhao, W. X., Zhou, K., Li, J., Tang, T., Wang, X., Hou, Y., Min, Y., Zhang, B., Zhang, J., Dong, Z., et al · 2023

Later among the works it cites.

A survey on model compression for large language models

Zhu, X., Li, J., Liu, Y., Ma, C., and Wang, W · 2023

Later among the works it cites.