Fetching the paper…

Reading the bibliography…

2024

LoRA-XS: Low-Rank Adaptation with Extremely Small Number of Parameters

Bałazy, Klaudia, Banaei, Mohammadreza, Aberer, Karl et al.

Original
PDF
Post on X

Understand

The growth of large language models underscores the need for parameter-efficient fine-tuning.

Despite its popularity, LoRA encounters storage and computational challenges when deploying multiple task- or user-specific modules.
To address this, we introduce LoRA-XS, a novel fine-tuning method backed by a theoretical derivation.
LoRA-XS drastically reduces trainable parameters by incorporating a small, trainable weight matrix between frozen low-rank matrices derived from the Singular Value Decomposition of pre-trained weights.

LoRA-XS: Low-Rank Adaptation with Extremely Small Number of Parameters · Around

Built on

Finding structure with randomness: Probabilistic algorithms for constructing approximate matrix decompositions
N. Halko, P.-G. Martinsson, and J. A. Tropp · 2011
Earlier work this paper cites.
Decoupled weight decay regularization
Original
I. Loshchilov and F. Hutter · 2017
Earlier work this paper cites.
Attention is all you need
A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin · 2017
Earlier work this paper cites.
Think you have solved question answering? try arc, the ai2 reasoning challenge
Original
P. Clark, I. Cowhey, O. Etzioni, T. Khot, A. Sabharwal, C. Schoenick, and O. Tafjord · 2018
Earlier work this paper cites.
Can a suit of armor conduct electricity? a new dataset for open book question answering
Original
T. Mihaylov, P. Clark, T. Khot, and A. Sabharwal · 2018
Earlier work this paper cites.
Glue: A multi-task benchmark and analysis platform for natural language understanding
Original
A. Wang, A. Singh, J. Michael, F. Hill, O. Levy, and S. R. Bowman · 2018
Earlier work this paper cites.
Boolq: Exploring the surprising difficulty of natural yes/no questions
Original
C. Clark, K. Lee, M.-W. Chang, T. Kwiatkowski, M. Collins, and K. Toutanova · 2019
Earlier work this paper cites.
Parameter-efficient transfer learning for nlp
N. Houlsby, A. Giurgiu, S. Jastrzebski, B. Morrone, Q. De Laroussilhe, A. Gesmundo, M. Attariyan, and S. Gelly · 2019
Earlier work this paper cites.
Roberta: A robustly optimized bert pretraining approach
Original
Y. Liu, M. Ott, N. Goyal, J. Du, M. Joshi, D. Chen, O. Levy, M. Lewis, L. Zettlemoyer, and V. Stoyanov · 2019
Earlier work this paper cites.
Pytorch: An imperative style, high-performance deep learning library
A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, et al · 2019
Earlier work this paper cites.
Socialiqa: Commonsense reasoning about social interactions
Original
M. Sap, H. Rashkin, D. Chen, R. LeBras, and Y. Choi · 2019
Earlier work this paper cites.
Huggingface’s transformers: State-of-the-art natural language processing
Original
T. Wolf, L. Debut, V. Sanh, J. Chaumond, C. Delangue, A. Moi, P. Cistac, T. Rault, R. Louf, M. Funtowicz, et al · 2019
Earlier work this paper cites.
Hellaswag: Can a machine really finish your sentence?
Original
R. Zellers, A. Holtzman, Y. Bisk, A. Farhadi, and Y. Choi · 2019
Earlier work this paper cites.
Piqa: Reasoning about physical commonsense in natural language
Y. Bisk, R. Zellers, J. Gao, Y. Choi, et al · 2020
Earlier work this paper cites.

Similar

Language models are few-shot learners
T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, et al · 2020
Cited alongside, same era.
Adapterfusion: Non-destructive task composition for transfer learning
Original
J. Pfeiffer, A. Kamath, A. Rücklé, K. Cho, and I. Gurevych · 2020
Cited alongside, same era.
Training verifiers to solve math word problems
Original
K. Cobbe, V. Kosaraju, M. Bavarian, M. Chen, H. Jun, L. Kaiser, M. Plappert, J. Tworek, J. Hilton, R. Nakano, et al · 2021
Cited alongside, same era.
Measuring mathematical problem solving with the math dataset
Original
D. Hendrycks, C. Burns, S. Kadavath, A. Arora, S. Basart, E. Tang, D. Song, and J. Steinhardt · 2021
Cited alongside, same era.
Lora: Low-rank adaptation of large language models
Original

Then

Vera: Vector-based random matrix adaptation
Original
D. J. Kopiczko, T. Blankevoort, and Y. M. Asano · 2023
Later among the works it cites.
Loftq: Lora-fine-tuning-aware quantization for large language models
Original
Y. Li, Y. Yu, C. Liang, P. He, N. Karampatziakis, W. Chen, and T. Zhao · 2023
Later among the works it cites.
Gpt understands, too
X. Liu, Y. Zheng, Z. Du, M. Ding, Y. Qian, Z. Yang, and J. Tang · 2023
Later among the works it cites.
Tied-lora: Enhacing parameter efficiency of lora with weight tying
Original
A. Renduchintala, T. Konuk, and O. Kuchaiev · 2023
Later among the works it cites.
Llama 2: Open foundation and fine-tuned chat models
Original
H. Touvron, L. Martin, K. Stone, P. Albert, A. Almahairi, Y. Babaei, N. Bashlykov, S. Batra, P. Bhargava, S. Bhosale, et al · 2023

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

E. J. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, and W. Chen · 2021

Cited alongside, same era.

The power of scale for parameter-efficient prompt tuning

Original

B. Lester, R. Al-Rfou, and N. Constant · 2021

Cited alongside, same era.

Prefix-tuning: Optimizing continuous prompts for generation

X. L. Li and P. Liang · 2021

Cited alongside, same era.

Winogrande: An adversarial winograd schema challenge at scale

K. Sakaguchi, R. L. Bras, C. Bhagavatula, and Y. Choi · 2021

Cited alongside, same era.

Bitfit: Simple parameter-efficient fine-tuning for transformer-based masked language-models

E. B. Zaken, S. Ravfogel, and Y. Goldberg · 2021

Cited alongside, same era.

Attempt: Parameter-efficient multi-task tuning via attentional mixtures of soft prompts

Original

A. Asai, M. Salehi, M. E. Peters, and H. Hajishirzi · 2022

Cited alongside, same era.

Few-shot parameter-efficient fine-tuning is better and cheaper than in-context learning

H. Liu, D. Tam, M. Muqeeth, J. Mohta, T. Huang, M. Bansal, and C. A. Raffel · 2022

Cited alongside, same era.

Peft: State-of-the-art parameter-efficient fine-tuning methods

S. Mangrulkar, S. Gugger, L. Debut, Y. Belkada, S. Paul, and B. Bossan · 2022

Cited alongside, same era.

Later among the works it cites.

Metamath: Bootstrap your own mathematical questions for large language models

L. Yu, W. Jiang, H. Shi, J. Yu, Z. Liu, Y. Zhang, J. T. Kwok, Z. Li, A. Weller, and W. Liu · 2023

Later among the works it cites.

Adaptive budget allocation for parameter-efficient fine-tuning

Q. Zhang, M. Chen, A. Bukharin, P. He, Y. Cheng, W. Chen, and T. Zhao · 2023

Later among the works it cites.

Qlora: Efficient finetuning of quantized llms

T. Dettmers, A. Pagnoni, A. Holtzman, and L. Zettlemoyer · 2024

Closest in time.

The llama 3 herd of models

A. Dubey, A. Jauhri, A. Pandey, A. Kadian, A. Al-Dahle, A. Letman, A. Mathur, A. Schelten, A. Yang, A. Fan, et al · 2024

Closest in time.

Lora+: Efficient low rank adaptation of large models

S. Hayou, N. Ghosh, and B. Yu · 2024

Closest in time.

Dora: Weight-decomposed low-rank adaptation

S.-Y. Liu, C.-Y. Wang, H. Yin, P. Molchanov, Y.-C. F. Wang, K.-T. Cheng, and M.-H. Chen · 2024

Closest in time.

Pissa: Principal singular values and singular vectors adaptation of large language models

F. Meng, Z. Wang, and M. Zhang · 2024

Closest in time.

Gemma: Open models based on gemini research and technology

G. Team, T. Mesnard, C. Hardin, R. Dadashi, S. Bhupatiraju, S. Pathak, L. Sifre, M. Rivière, M. S. Kale, J. Love, et al · 2024

Closest in time.