Fetching the paper…

Reading the bibliography…

2024

Low-Rank Adaptation for Foundation Models: A Comprehensive Review

Yang, Menglin, Chen, Jialin, Tao, Jinkai et al.

Original
PDF
Post on X

Understand

The rapid advancement of foundation modelslarge-scale neural networks trained on diverse, extensive datasetshas revolutionized artificial intelligence, enabling unprecedented advancements across domains such as natural language processing, computer vision, and scientific discovery.

However, the substantial parameter count of these models, often reaching billions or trillions, poses significant challenges in adapting them to specific downstream tasks.
Low-Rank Adaptation (LoRA) has emerged as a highly promising approach for mitigating these challenges, offering a parameter-efficient mechanism to fine-tune foundation models with minimal computational overhead.
This survey provides the first comprehensive review of LoRA techniques beyond large Language Models to general foundation models, including recent techniques foundations, emerging frontiers and applications of low-rank adaptation across multiple domains.

Low-Rank Adaptation for Foundation Models: A Comprehensive Review · Around

Built on

N. Kumari, B. Zhang, R. Zhang, E. Shechtman, and J.-Y. Zhu, “Multi-concept customization of text-to-image diffusion,” in CVPR , 2023, pp. 1931–1941
1941
Earlier work this paper cites.
S. Dou, E. Zhou, Y. Liu, S. Gao, W. Shen, L. Xiong, Y. Zhou, X. Wang, Z. Xi, X. Fan et al. , “LoRAMoE: Alleviating world knowledge forgetting in large language models via moe-style plugin,” in ACL , 2024, pp. 1932–1945
1945
Earlier work this paper cites.
N. Hansen and A. Ostermeier, “Adapting arbitrary normal mutation distributions in evolution strategies: The covariance matrix adaptation,” in ICEC . IEEE, 1996, pp. 312–317
1996
Earlier work this paper cites.
M. E. Wall, A. Rechtsteiner, and L. M. Rocha, “Singular value decomposition and principal component analysis,” in A practical approach to microarray data analysis . Springer, 2003, pp. 91–109
2003
Earlier work this paper cites.
I. V. Oseledets, “Tensor-train decomposition,” SISC , vol. 33, no. 5, pp. 2295–2317, 2011
2011
Earlier work this paper cites.
A. Graves, “Sequence transduction with recurrent neural networks,” arXiv:1211.3711 , 2012
Original
2012
Earlier work this paper cites.
D. P. Kingma, “Adam: A method for stochastic optimization,” arXiv:1412.6980 , 2014
Original
2014
Earlier work this paper cites.
T. Bolukbasi, K.-W. Chang, J. Y. Zou, V. Saligrama, and A. T. Kalai, “Man is to computer programmer as woman is to homemaker? debiasing word embeddings,” in NeurIPS , vol. 29, 2016
2016
Earlier work this paper cites.
S. Lacoste-Julien, “Convergence rate of frank-wolfe for non-convex objectives,” arXiv:1607.00345 , 2016
Original
2016
Earlier work this paper cites.
I. Hubara, M. Courbariaux, D. Soudry, R. El-Yaniv, and Y. Bengio, “Quantized neural networks: Training neural networks with low precision weights and activations,” in JMLR , 2017, pp. 6869–6898
2017
Earlier work this paper cites.
P. Micikevicius, S. Narang, J. Alben, G. Diamos, E. Elsen, D. Garcia, B. Ginsburg, M. Houston, O. Kuchaiev, G. Venkatesh et al. , “Mixed precision training,” arXiv:1710.03740 , 2017
Original
2017
Earlier work this paper cites.
N. Shazeer, A. Mirhoseini, K. Maziarz, A. Davis, Q. Le, G. Hinton, and J. Dean, “Outrageously large neural networks: The sparsely-gated mixture-of-experts layer,” arXiv:1701.06538 , 2017
Original
2017
Earlier work this paper cites.
C. Li, H. Farkhoor, R. Liu, and J. Yosinski, “Measuring the intrinsic dimension of objective landscapes,” arXiv:1804.08838 , 2018
Original
2018
Earlier work this paper cites.
J. Devlin, “Bert: Pre-training of deep bidirectional transformers for language understanding,” arXiv:1810.04805 , 2018
Original
2018
Earlier work this paper cites.
Y. Liu, M. Ott, N. Goyal, J. Du, M. Joshi, D. Chen, O. Levy, M. Lewis, L. Zettlemoyer, and V. Stoyanov, “Roberta: A robustly optimized bert pretraining approach,” arXiv:1907.11692 , 2019
Original
2019
Earlier work this paper cites.
N. Houlsby, A. Giurgiu, S. Jastrzebski, B. Morrone, Q. De Laroussilhe, A. Gesmundo, M. Attariyan, and S. Gelly, “Parameter-efficient transfer learning for nlp,” in ICML , 2019
2019
Earlier work this paper cites.
R. Banner, Y. Nahshan, and D. Soudry, “Post training 4-bit quantization of convolutional networks for rapid-deployment,” in NeurIPS , vol. 32, 2019
2019
Earlier work this paper cites.
T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell et al. , “Language models are few-shot learners,” in NeurIPS , vol. 33, 2020
2020
Earlier work this paper cites.
A. Baevski, Y. Zhou, A. Mohamed, and M. Auli, “wav2vec 2.0: A framework for self-supervised learning of speech representations,” in NeurIPS , vol. 33, 2020
2020
Earlier work this paper cites.
S. Chithrananda, G. Grand, and B. Ramsundar, “Chemberta: Large-scale self-supervised pretraining for molecular property prediction,” arXiv:2010.09885 , 2020
Original
2020
Earlier work this paper cites.
J. Kaplan, S. McCandlish, T. Henighan, T. B. Brown, B. Chess, R. Child, S. Gray, A. Radford, J. Wu, and D. Amodei, “Scaling laws for neural language models,” arXiv:2001.08361 , 2020
Original
2020
Earlier work this paper cites.
S. Rajbhandari, J. Rasley, O. Ruwase, and Y. He, “Zero: Memory optimizations toward training trillion parameter models,” in SC20 . IEEE, 2020, pp. 1–16
2020
Earlier work this paper cites.
A. Aghajanyan, L. Zettlemoyer, and S. Gupta, “Intrinsic dimensionality explains the effectiveness of language model fine-tuning,” arXiv:2012.13255 , 2020
Original
2020
Earlier work this paper cites.
C. He, S. Li, J. So, X. Zeng, M. Zhang, H. Wang, X. Wang, P. Vepakomma, A. Singh, H. Qiu et al. , “Fedml: A research library and benchmark for federated machine learning,” arXiv:2007.13518 , 2020
Original
2020
Earlier work this paper cites.
W. Y. B. Lim, N. C. Luong, D. T. Hoang, Y. Jiao, Y.-C. Liang, Q. Yang, D. Niyato, and C. Miao, “Federated learning in mobile edge networks: A comprehensive survey,” IEEE communications surveys & tutorials , vol. 22, no. 3, pp. 2031–2063, 2020
2020
Earlier work this paper cites.
D. Alexey, “An image is worth 16x16 words: Transformers for image recognition at scale,” arXiv: 2010.11929 , 2020
2020
Earlier work this paper cites.
J. Ho, A. Jain, and P. Abbeel, “Denoising diffusion probabilistic models,” in NeurIPS , vol. 33, 2020
2020
Earlier work this paper cites.
R. Bommasani, D. A. Hudson, E. Adeli, R. Altman, S. Arora, S. von Arx, M. S. Bernstein, J. Bohg, A. Bosselut, E. Brunskill et al. , “On the opportunities and risks of foundation models,” arXiv:2108.07258 , 2021
Original
2021
Earlier work this paper cites.
Z. Liu, Y. Lin, Y. Cao, H. Hu, Y. Wei, Z. Zhang, S. Lin, and B. Guo, “Swin transformer: Hierarchical vision transformer using shifted windows,” in ICCV , 2021, pp. 10 012–10 022
2021
Earlier work this paper cites.
J. Jumper, R. Evans, A. Pritzel, T. Green, M. Figurnov, O. Ronneberger, K. Tunyasuvunakool, R. Bates, A. Žídek, A. Potapenko et al. , “Highly accurate protein structure prediction with alphafold,” Nature , vol. 596, no. 7873, pp. 583–589, 2021
2021
Earlier work this paper cites.
A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark et al. , “Learning transferable visual models from natural language supervision,” in ICML , 2021
2021
Earlier work this paper cites.
E. J. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, and W. Chen, “Lora: Low-rank adaptation of large language models,” arXiv:2106.09685 , 2021
Original
2021
Earlier work this paper cites.
E. M. Bender, T. Gebru, A. McMillan-Major, and S. Shmitchell, “On the dangers of stochastic parrots: Can language models be too big?” Proc. 2021 ACM Conf. Fairness, Accountability, and Transparency , pp. 610–623, 2021
2021
Earlier work this paper cites.
L. Weidinger, J. Uesato, M. Rauh, C. Griffin et al. , “Ethical and social risks of harm from language models,” arXiv:2112.04359 , 2021
Original
2021
Earlier work this paper cites.
J. Wei, M. Bosma, V. Y. Zhao, K. Guu, A. W. Yu, B. Lester, N. Du, A. M. Dai, and Q. V. Le, “Finetuned language models are zero-shot learners,” arXiv:2109.01652 , 2021
Original
2021
Earlier work this paper cites.
E. B. Zaken, S. Ravfogel, and Y. Goldberg, “Bitfit: Simple parameter-efficient fine-tuning for transformer-based masked language-models,” arXiv:2106.10199 , 2021
Original
2021
Earlier work this paper cites.
A. Gholami, Z. Kim, K. Dong, S. Yao, M. W. Mahoney, and K. Keutzer, “A survey of quantization methods for efficient neural network inference,” arXiv:2103.13630 , 2021
Original
2021
Earlier work this paper cites.
S. Zuo, X. Liu, J. Jiao, Y. J. Kim, H. Hassan, R. Zhang, T. Zhao, and J. Gao, “Taming sparsely activated transformer with stochastic experts,” arXiv:2110.04260 , 2021
Original
2021
Earlier work this paper cites.
P. He, X. Liu, J. Gao, and W. Chen, “Deberta: Decoding-enhanced bert with disentangled attention,” arXiv:2006.03654 , 2021
Original
2021
Earlier work this paper cites.
L. Yuan, D. Chen, Y.-L. Chen, N. Codella, X. Dai, J. Gao, H. Hu, X. Huang, B. Li, C. Li et al. , “Florence: A new foundation model for computer vision,” arXiv:2111.11432 , 2021
Original
2021
Earlier work this paper cites.
A. Elnaggar, M. Heinzinger, C. Dallago, G. Rihawi, Y. Wang, L. Jones, T. Gibbs, T. Feher, C. Angerer, M. Steinegger, D. Bhowmik, and B. Rost, “Prottrans: Towards cracking the language of life’s code through self-supervised deep learning and high performance computing,” 2021
2021
Earlier work this paper cites.
B. Lim and S. Zohren, “Time-series forecasting with deep learning: a survey,” Philosophical Transactions of the Royal Society A , vol. 379, no. 2194, p. 20200209, 2021
2021
Earlier work this paper cites.
K. He, X. Chen, S. Xie, Y. Li, P. Dollár, and R. Girshick, “Masked autoencoders are scalable vision learners,” in CVPR , 2022, pp. 16 000–16 009
2022
Earlier work this paper cites.
A. Radford, J. W. Kim, T. Xu, G. Brockman, C. McLeavey, and I. Sutskever, “Robust speech recognition via large-scale weak supervision,” arXiv:2212.04356 , 2022
Original
2022
Earlier work this paper cites.
R. Rombach, A. Blattmann, D. Lorenz, P. Esser, and B. Ommer, “High-resolution image synthesis with latent diffusion models,” CVPR , pp. 10 684–10 695, 2022
2022
Earlier work this paper cites.
A. Ramesh, P. Dhariwal, A. Nichol, C. Chu, and M. Chen, “Hierarchical text-conditional image generation with clip latents,” arXiv:2204.06125 , 2022
Original
2022
Earlier work this paper cites.
H. Liu, D. Tam, M. Muqeeth, J. Mohta, T. Huang, M. Bansal, and C. A. Raffel, “Few-shot parameter-efficient fine-tuning is better and cheaper than in-context learning,” in NeurIPS , vol. 35, 2022
2022
Earlier work this paper cites.
J. Li, D. Li, C. Xiong, and S. Hoi, “Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation,” in ICML , 2022
2022
Earlier work this paper cites.
S. He, L. Ding, D. Dong, M. Zhang, and D. Tao, “Sparseadapter: An easy approach for improving the parameter-efficiency of adapters,” arXiv:2210.04284 , 2022
Original
2022
Earlier work this paper cites.
W. Fedus, B. Zoph, and N. Shazeer, “Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity,” JMLR , vol. 23, no. 120, pp. 1–39, 2022
2022
Earlier work this paper cites.
S. Horvóth, C.-Y. Ho, L. Horvath, A. N. Sahu, M. Canini, and P. Richtárik, “Natural compression for distributed deep learning,” in Mathematical and Scientific Machine Learning . PMLR, 2022, pp. 129–141
2022
Earlier work this paper cites.
Z. Lin, H. Akin, R. Rao, B. Hie, Z. Zhu, W. Lu, A. dos Santos Costa, M. Fazel-Zarandi, T. Sercu, S. Candido et al. , “Language models of protein sequences at the scale of evolution enable accurate structure prediction,” BioRxiv , vol. 2022, p. 500902, 2022
2022
Earlier work this paper cites.
M. Sun, K. Zhou, X. He, Y. Wang, and X. Wang, “Gppt: Graph pre-training and prompt tuning to generalize graph neural networks,” in KDD , 2022, pp. 1717–1727
2022
Earlier work this paper cites.
Y.-L. Sung, J. Cho, and M. Bansal, “Vl-adapter: Parameter-efficient transfer learning for vision-and-language tasks,” in CVPR , 2022, pp. 5227–5237
2022
Earlier work this paper cites.
C. Zhou, Q. Li, C. Li, J. Yu, Y. Liu, G. Wang, K. Zhang, C. Ji, Q. Yan, L. He et al. , “A comprehensive survey on pretrained foundation models: A history from bert to chatgpt,” arXiv:2302.09419 , 2023
Original
2023
Earlier work this paper cites.
H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix, B. Rozière, N. Goyal, E. Hambro, F. Azhar et al. , “Llama: Open and efficient foundation language models,” arXiv:2302.13971 , 2023
Original
2023
Earlier work this paper cites.
A. Kirillov, E. Mintun, N. Ravi, H. Mao, C. Rolland, L. Gustafson, T. Xiao, S. Whitehead, A. C. Berg, W.-Y. Lo et al. , “Segment anything,” in ICCV , 2023, pp. 4015–4026
2023
Earlier work this paper cites.
Z. Lin, H. Akin, R. Rao, B. Hie, Z. Zhu, W. Lu, N. Smetanin, R. Verkuil, O. Kabeli, Y. Shmueli et al. , “Evolutionary-scale prediction of atomic level protein structure with a language model,” Science , vol. 379, no. 6637, pp. 1123–1130, 2023
2023
Earlier work this paper cites.
V. Lialin, V. Deshpande, and A. Rumshisky, “Scaling down to scale up: A guide to parameter-efficient fine-tuning,” arXiv:2303.15647 , 2023
Original
2023
Earlier work this paper cites.
Q. Zhang, M. Chen, A. Bukharin, N. Karampatziakis, P. He, Y. Cheng, W. Chen, and T. Zhao, “Adalora: Adaptive budget allocation for parameter-efficient fine-tuning,” arXiv:2303.10512 , 2023
Original
2023
Earlier work this paper cites.
N. Ding, X. Lv, Q. Wang, Y. Chen, B. Zhou, Z. Liu, and M. Sun, “Sparse low-rank adaptation of pre-trained language models,” arXiv:2311.11696 , 2023
Original
2023
Earlier work this paper cites.
L. Zhang, L. Zhang, S. Shi, X. Chu, and B. Li, “Lora-FA: Memory-efficient low-rank adaptation for large language models fine-tuning,” arXiv:2308.03303 , 2023
Original
2023
Earlier work this paper cites.
D. J. Kopiczko, T. Blankevoort, and Y. M. Asano, “Vera: Vector-based random matrix adaptation,” arXiv:2310.11454 , 2023
Original
2023
Earlier work this paper cites.
S. A. Koohpayegani, K. Navaneet, P. Nooralinejad, S. Kolouri, and H. Pirsiavash, “Nola: Networks as linear combination of low rank random basis,” arXiv:2310.02556 , 2023
Original
2023
Earlier work this paper cites.
A. Renduchintala, T. Konuk, and O. Kuchaiev, “Tied-lora: Enhacing parameter efficiency of lora with weight tying,” arXiv:2311.09578 , 2023
Original
2023
Earlier work this paper cites.
Y. Li, Y. Yu, C. Liang, P. He, N. Karampatziakis, W. Chen, and T. Zhao, “Loftq: Lora-fine-tuning-aware quantization for large language models,” arXiv:2310.08659 , 2023
Original
2023
Earlier work this paper cites.
Y. Xu, L. Xie, X. Gu, X. Chen, H. Chang, H. Zhang, Z. Chen, X. Zhang, and Q. Tian, “Qa-lora: Quantization-aware low-rank adaptation of large language models,” arXiv:2309.14717 , 2023
Original
2023
Earlier work this paper cites.
H. Guo, P. Greengard, E. P. Xing, and Y. Kim, “Lq-lora: Low-rank plus quantized matrix decomposition for efficient language model finetuning,” arXiv:2311.12023 , 2023
Original
2023
Earlier work this paper cites.
M. Valipour, M. Rezagholizadeh, I. Kobyzev, and A. Ghodsi, “DyLoRA: Parameter-efficient tuning of pre-trained models using dynamic search-free low-rank adaptation,” EACL , 2023
2023
Earlier work this paper cites.
V. Lialin, N. Shivagunde, S. Muckatira, and A. Rumshisky, “Relora: High-rank training through low-rank updates,” arXiv:2307.05695 , 2023
Original
2023
Earlier work this paper cites.
D. Kalajdzievski, “A rank stabilization scaling factor for fine-tuning with lora,” arXiv:2312.03732 , 2023
Original
2023
Earlier work this paper cites.
S. Malladi, A. Wettig, D. Yu, D. Chen, and S. Arora, “A kernel-based view of language model fine-tuning,” in ICML , 2023
2023
Earlier work this paper cites.
C. Huang, Q. Liu, B. Y. Lin, T. Pang, C. Du, and M. Lin, “Lorahub: Efficient cross-task generalization via dynamic lora composition,” arXiv:2307.13269 , 2023
Original
2023
Earlier work this paper cites.
Y. Wen and S. Chaudhuri, “Batched low-rank adaptation of foundation models,” arXiv:2312.05677 , 2023
Original
2023
Earlier work this paper cites.
S. Kim, H. Yang, Y. Kim, Y. Hong, and E. Park, “Hydra: Multi-head low-rank adaptation for parameter efficient fine-tuning,” arXiv:2309.06922 , 2023
Original
2023
Earlier work this paper cites.
A. Chavan, Z. Liu, D. Gupta, E. Xing, and Z. Shen, “One-for-all: Generalized lora for parameter-efficient fine-tuning,” arXiv:2306.07967 , 2023
Original
2023
Earlier work this paper cites.
T. Zadouri, A. Üstün, A. Ahmadian, B. Ermiş, A. Locatelli, and S. Hooker, “Pushing mixture of experts to the limit: Extremely parameter efficient moe for instruction tuning,” arXiv:2309.05444 , 2023
Original
2023
Earlier work this paper cites.
T. Chen, Z. Zhang, A. Jaiswal, S. Liu, and Z. Wang, “Sparse moe as the new dropout: Scaling dense and self-slimmable transformers,” arXiv:2303.01610 , 2023
Original
2023
Earlier work this paper cites.
Y. Gou, Z. Liu, K. Chen, L. Hong, H. Xu, A. Li, D.-Y. Yeung, J. T. Kwok, and Y. Zhang, “Mixture of cluster-conditional lora experts for vision-language instruction tuning,” arXiv:2312.12379 , 2023
Original
2023
Earlier work this paper cites.
X. Wang, T. Chen, Q. Ge, H. Xia, R. Bao, R. Zheng, Q. Zhang, T. Gui, and X. Huang, “Orthogonal subspace learning for language model continual learning,” arXiv:2310.14152 , 2023
Original
2023
Earlier work this paper cites.
M. Wistuba, P. T. Sivaprasad, L. Balles, and G. Zappella, “Continual learning with low rank adaptation,” arXiv:2311.17601 , 2023
Original
2023
Earlier work this paper cites.
R. Chitale, A. Vaidya, A. Kane, and A. Ghotkar, “Task arithmetic with lora for continual learning,” arXiv:2311.02428 , 2023
Original
2023
Earlier work this paper cites.
J. Chen and D. Yang, “Unlearn what you want to forget: Efficient unlearning for llms,” in EMNLP , 2023, pp. 12 041–12 052
2023
Earlier work this paper cites.
Y. Wang, Y. Lin, X. Zeng, and G. Zhang, “Privatelora for efficient privacy preserving llm,” arXiv:2311.14030 , 2023
Original
2023
Earlier work this paper cites.
S. Babakniya, A. R. Elkordy, Y. H. Ezzeldin, Q. Liu, K.-B. Song, M. El-Khamy, and S. Avestimehr, “Slora: Federated parameter efficient fine-tuning of language models,” arXiv:2308.06522 , 2023
Original
2023
Earlier work this paper cites.

Similar

L. Yi, H. Yu, G. Wang, and X. Liu, “pfedlora: Model-heterogeneous personalized federated learning with lora tuning,” arXiv:2310.13283 , 2023
Original
2023
Cited alongside, same era.
Y. J. Cho, L. Liu, Z. Xu, A. Fahrezi, M. Barnes, and G. Joshi, “Heterogeneous lora for federated fine-tuning of on-device foundation models,” in NeurIPS Workshop , 2023
2023
Cited alongside, same era.
Y. Chen, S. Qian, H. Tang, X. Lai, Z. Liu, S. Han, and J. Jia, “Longlora: Efficient fine-tuning of long-context large language models,” arXiv:2309.12307 , 2023
Original
2023
Cited alongside, same era.
J. Yang, “Longqlora: Efficient and effective method to extend context length of large language models,” arXiv:2311.04879 , 2023
Original
2023
Cited alongside, same era.
S. Chen, S. Wong, L. Chen, and Y. Tian, “Extending context window of large language models via positional interpolation,” arXiv:2306.15595 , 2023

Then

W. Shi, R. Xu, Y. Zhuang, Y. Yu, H. Wu, C. Yang, and M. D. Wang, “Medadapter: Efficient test-time adaptation of large language models towards medical reasoning,” arXiv:2405.03000 , 2024
Original
2024
Closest in time.
C. Christophe, P. K. Kanithi, P. Munjal, T. Raha, N. Hayat, R. Rajan, A. Al-Mahrooqi, A. Gupta, M. U. Salman, G. Gosal et al. , “Med42–evaluating fine-tuning strategies for medical llms: Full-parameter vs. parameter-efficient approaches,” arXiv:2404.14779 , 2024
Original
2024
Closest in time.
Z. Li, N. Pang, and X. Zhao, “Instruction tuning large language models for multimodal relation extraction using lora,” in WISA . Springer, 2024, pp. 364–376
2024
Closest in time.
Y. Zhang, J. Wang, L.-C. Yu, D. Xu, and X. Zhang, “Personalized lora for human-centered text understanding,” in AAAI , vol. 38, 2024, pp. 19 588–19 596
2024
Closest in time.
S. Khanna, M. Irgau, D. B. Lobell, and S. Ermon, “Explora: Parameter-efficient extended pre-training to adapt vision transformers under domain shifts,” arXiv:2406.10973 , 2024

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

2023

Cited alongside, same era.

Y. Sheng, S. Cao, D. Li, C. Hooper, N. Lee, S. Yang, C. Chou, B. Zhu, L. Zheng, K. Keutzer et al. , “S-LoRA: Serving thousands of concurrent lora adapters,” arXiv:2311.03285 , 2023

2023

Cited alongside, same era.

Q. Liu, X. Wu, X. Zhao, Y. Zhu, D. Xu, F. Tian, and Y. Zheng, “Moelora: An moe-based parameter efficient fine-tuning method for multi-task medical applications,” CoRR , 2023

2023

Cited alongside, same era.

Z. Xiao, W. Held, Y. Liu, and D. Yang, “Task-agnostic low-rank adapters for unseen english dialects,” arXiv:2311.00915 , 2023

Original

2023

Cited alongside, same era.

Y. Zhang, M. Wang, P. Tiwari, Q. Li, B. Wang, and J. Qin, “Dialoguellm: Context and emotion knowledge-tuned llama models for emotion recognition in conversations,” arXiv:2310.11374 , 2023

2023

Cited alongside, same era.

M. Oquab, T. Darcet, T. Moutakanni, H. Vo, M. Szafraniec, V. Khalidov, P. Fernandez, D. Haziza, F. Massa, A. El-Nouby et al. , “Dinov2: Learning robust visual features without supervision,” arXiv:2304.07193 , 2023

2023

Cited alongside, same era.

C. Kong, H. Li, and S. Wang, “Enhancing general face forgery detection via vision transformer with low-rank adaptation,” arXiv:2303.00917 , 2023

2023

Cited alongside, same era.

K. Zhang and D. Liu, “Customized segment anything model for medical image segmentation,” arXiv:2304.13785 , 2023

2023

Cited alongside, same era.

2024

Closest in time.

Y. Zhu, Z. Shen, Z. Zhao, S. Wang, X. Wang, X. Zhao, D. Shen, and Q. Wang, “Melo: Low-rank adaptation is better than fine-tuning for medical image diagnosis,” in ISBI . IEEE, 2024, pp. 1–5

2024

Closest in time.

S. Aleem, J. Dietlmeier, E. Arazo, and S. Little, “Convlora and adabn based domain adaptation via self-training,” arXiv:2402.04964 , 2024

2024

Closest in time.

W. Yue, J. Zhang, K. Hu, Y. Xia, J. Luo, and Z. Wang, “Surgicalsam: Efficient class promptable surgical instrument segmentation,” in AAAI , vol. 38, 2024, pp. 6890–6898

2024

Closest in time.

Z. Liu, H. Wang, Y. Kang, and S. Wang, “Mixture of low-rank experts for transferable ai-generated image detection,” arXiv:2404.04883 , 2024

2024

Closest in time.

C. Kong, A. Luo, P. Bao, Y. Yu, H. Li, Z. Zheng, S. Wang, and A. C. Kot, “Moe-ffd: Mixture of experts for generalized and parameter-efficient face forgery detection,” IEEE Trans. Dependable Secure Comput. , 2024

2024

Closest in time.

Z. Yuan, J. Zhang, and S. Shan, “Fulllora-at: Efficiently boosting the robustness of pretrained vision transformers,” arXiv:2401.01752 , 2024

2024

Closest in time.

L. Lin, H. Fan, Z. Zhang, Y. Wang, Y. Xu, and H. Ling, “Tracking meets lora: Faster training, larger model, stronger performance,” in ECCV . Springer, 2024

2024

Closest in time.

Y. Frenkel, Y. Vinker, A. Shamir, and D. Cohen-Or, “Implicit style-content separation using b-lora,” arXiv:2403.14572 , 2024

2024

Closest in time.

S. Borse, S. Kadambi, N. P. Pandey, K. Bhardwaj, V. Ganapathy, S. Priyadarshi, R. Garrepalli, R. Esteves, M. Hayat, and F. Porikli, “Foura: Fourier low rank adaptation,” arXiv:2406.08798 , 2024

2024

Closest in time.

V. Shah, N. Ruiz, F. Cole, E. Lu, S. Lazebnik, Y. Li, and V. Jampani, “Ziplora: Any subject in any style by effectively merging loras,” in ECCV . Springer, 2024, pp. 422–438

2024

Closest in time.

Y. Gu, X. Wang, J. Z. Wu, Y. Shi, Y. Chen, Z. Fan, W. Xiao, R. Zhao, S. Chang, W. Wu et al. , “Mix-of-show: Decentralized low-rank adaptation for multi-concept customization of diffusion models,” in NeurIPS , vol. 36, 2024

2024

Closest in time.

M. Zhong, Y. Shen, S. Wang, Y. Lu, Y. Jiao, S. Ouyang, D. Yu, J. Han, and W. Chen, “Multi-lora composition for image generation,” arXiv:2402.16843 , 2024

2024

Closest in time.

Y. Yang, W. Wang, L. Peng, C. Song, Y. Chen, H. Li, X. Yang, Q. Lu, D. Cai, B. Wu, and W. Liu, “Lora-composer: Leveraging low-rank adaptation for multi-concept customization in training-free diffusion models,” arXiv:2403.11627 , 2024

2024

Closest in time.

R. Po, G. Yang, K. Aberman, and G. Wetzstein, “Orthogonal adaptation for modular customization of diffusion models,” arXiv:2312.02432 , 2024

2024

Closest in time.

J. Zhu, Y. Chen, M. Ding, P. Luo, L. Wang, and J. Wang, “MoLE: Enhancing human-centric text-to-image diffusion via mixture of low-rank experts,” in NeurIPS , 2024

2024

Closest in time.

J. Cheng, P. Xie, X. Xia, J. Li, J. Wu, Y. Ren, H. Li, X. Xiao, M. Zheng, and L. Fu, “Resadapter: Domain consistent resolution adapter for diffusion models,” arXiv:2403.02084 , 2024

2024

Closest in time.

Y. Wang, Y. Zhang, and Z. Wang, “Fit: Flexible diffusion models for arbitrary image resolutions and aspect ratios,” arXiv:2402.12376 , 2024

2024

Closest in time.

T. Xu, K. Huang, P. Guo, Y. Zhou, L. Huang, H. Xue, and L. Xie, “Towards rehearsal-free multilingual asr: A lora-based case study on whisper,” arXiv:2408.10680 , 2024

2024

Closest in time.

Z. Song, J. Zhuo, Y. Yang, Z. Ma, S. Zhang, and X. Chen, “Lora-whisper: Parameter-efficient and extensible multilingual asr,” arXiv:2406.06619 , 2024

2024

Closest in time.

J. Kumar and S. Chimalakonda, “Code summarization without direct access to code-towards exploring federated llms for software engineering,” in EASE , 2024, pp. 100–109

2024

Closest in time.

F. Cui, H. Ren, Z. Guo, Y. Zhao, Y. Jiang, Y. Jiang, Y. Ding, X. Wang, and W. Qian, “Origen: Enhancing rtl code generation with code-to-code augmentation and self-reflection,” arXiv:2407.16237 , 2024

2024

Closest in time.

S. Zeng, D. Wang, L. Jiang, and D. Xu, “Parameter-efficient fine-tuning on large protein language models improves signal peptide prediction,” Genome research , vol. 34, 2024

2024

Closest in time.

R. Schmirler, M. Heinzinger, and B. Rost, “Fine-tuning protein language models boosts predictions across diverse tasks,” Nature Communications , vol. 15, no. 7407, 2024

2024

Closest in time.

L. Lv, Z. Lin, H. Li, Y. Liu, J. Cui, C. Yu-Chian Chen, L. Yuan, and Y. Tian, “Prollama: A protein large language model for multi-task protein language processing,” arXiv e-prints , pp. arXiv–2402, 2024

2024

Closest in time.

J. Dagdelen, A. Dunn, S. Lee, N. Walker, A. S. Rosen, G. Ceder, K. A. Persson, and A. Jain, “Structured information extraction from scientific text with large language models,” Nature communications , p. 1418, 2024

2024

Closest in time.

Z. Yang, H. Gao, D. Gao, L. Yang, L. Yang, X. Cai, W. Ning, and G. Zhang, “Mlora: Multi-domain low-rank adaptive network for ctr prediction,” in RecSys , 2024, pp. 287–297

2024

Closest in time.

J. Zhu, J. Lin, X. Dai, B. Chen, R. Shan, J. Zhu, R. Tang, Y. Yu, and W. Zhang, “Lifelong personalized low-rank adaptation of large language models for recommendation,” arXiv:2408.03533 , 2024

2024

Closest in time.

Z. Qin, “Atflrec: A multimodal recommender system with audio-text fusion and low-rank adaptation via instruction-tuned large language model,” arXiv:2409.08543 , 2024

2024

Closest in time.

Z. Zheng, W. Chao, Z. Qiu, H. Zhu, and H. Xiong, “Harnessing large language models for text-rich sequential recommendation,” in ACM WebConf , 2024, pp. 3207–3216

2024

Closest in time.

X. Kong, J. Wu, A. Zhang, L. Sheng, H. Lin, X. Wang, and X. He, “Customizing language models with instance-wise lora for sequential recommendation,” arXiv:2408.10159 , 2024

2024

Closest in time.

J. Ji, Z. Li, S. Xu, W. Hua, Y. Ge, J. Tan, and Y. Zhang, “Genrec: Large language model for generative recommendation,” in ECIR . Springer, 2024, pp. 494–502

2024

Closest in time.

Z.-R. Yang, J. Han, C.-D. Wang, and H. Liu, “Graphlora: Structure-aware contrastive low-rank adaptation for cross-graph transfer learning,” arXiv:2409.16670 , 2024

2024

Closest in time.

J. Liu, W. Ke, P. Wang, J. Wang, J. Gao, Z. Shang, G. Li, Z. Xu, K. Ji, and Y. Li, “Fast and continual knowledge graph embedding via incremental lora,” arXiv:2407.05705 , 2024

2024

Closest in time.

W. Ruan, W. Chen, X. Dang, J. Zhou, W. Li, X. Liu, and Y. Liang, “Low-rank adaptation for spatio-temporal forecasting,” arXiv:2404.07919 , 2024

2024

Closest in time.

T. Nie, Y. Mei, G. Qin, J. Sun, and W. Ma, “Channel-aware low-rank adaptation in time series forecasting,” arXiv:2407.17246 , 2024

2024

Closest in time.

D. Gupta, A. Bhatti, S. Parmar, C. Dan, Y. Liu, B. Shen, and S. Lee, “Low-rank adaptation of time series foundational models for out-of-domain modality forecasting,” arXiv:2405.10216 , 2024

2024

Closest in time.

G. Woo, C. Liu, A. Kumar, C. Xiong, S. Savarese, and D. Sahoo, “Unified training of universal time series forecasting transformers,” arXiv:2402.02592 , 2024

2024

Closest in time.

A. F. Ansari, L. Stella, C. Turkmen, X. Zhang, P. Mercado, H. Shen, O. Shchur, S. S. Rangapuram, S. Pineda Arango, S. Kapoor et al. , “Chronos: Learning the language of time series,” arXiv:2403.07815 , 2024

2024

Closest in time.

Y. Ren, Y. Chen, S. Liu, B. Wang, H. Yu, and Z. Cui, “TPLLM: A traffic prediction framework based on pretrained large language models,” arXiv:2403.02221 , 2024

2024

Closest in time.

Y. Ji, Y. Liu, Z. Zhang, Z. Zhang, Y. Zhao, G. Zhou, X. Zhang, X. Liu, and X. Zheng, “Advlora: Adversarial low-rank adaptation of vision-language models,” arXiv:2404.13425 , 2024

2024

Closest in time.

Z. Zong, B. Ma, D. Shen, G. Song, H. Shao, D. Jiang, H. Li, and Y. Liu, “Mova: Adapting mixture of vision experts to multimodal context,” arXiv:2404.13046 , 2024

2024

Closest in time.

J. Li, J. Cho, Y.-L. Sung, J. Yoon, and M. Bansal, “Selma: Learning and merging skill-specific text-to-image experts with auto-generated data,” arXiv:2403.06952 , 2024

2024

Closest in time.

S. Lu, Z. Wang, L. Li, Y. Liu, and A. W.-K. Kong, “Mace: Mass concept erasure in diffusion models,” arXiv:2403.06135 , 2024

2024

Closest in time.

H. Wang, X. Xiang, Y. Fan, and J.-H. Xue, “Customizing 360-degree panoramas through text-to-image diffusion models,” in WACV , 2024, pp. 4933–4943

2024

Closest in time.

L. Li, H. Zeng, C. Yang, H. Jia, and D. Xu, “Block-wise lora: Revisiting fine-grained lora for effective personalization and stylization in text-to-image generation,” arXiv:2403.07500 , 2024

2024

Closest in time.

Y. Fathullah, C. Wu, E. Lakomkin, J. Jia, Y. Shangguan, K. Li, J. Guo, W. Xiong, J. Mahadeokar, O. Kalinli et al. , “Prompting large language models with speech recognition abilities,” in ICASSP . IEEE, 2024, pp. 13 351–13 355

2024

Closest in time.

B. Yusuf, M. K. Baskar, A. Rosenberg, and B. Ramabhadran, “Speculative speech recognition by audio-prefixed low-rank adaptation of language models,” arXiv:2407.04641 , 2024

2024

Closest in time.

S. Palaskar, O. Rudovic, S. Dharur, F. Pesce, G. Krishna, A. Sivaraman, J. Berkowitz, A. H. Abdelaziz, S. Adya, and A. Tewfik, “Multimodal large language models with fusion low rank adaptation for device directed speech detection,” arXiv:2406.09617 , 2024

2024

Closest in time.

J. Liu, G. Li, J. Zhang, H. Dinkel, Y. Wang, Z. Yan, Y. Wang, and B. Wang, “Enhancing automated audio captioning via large language models with optimized audio encoding,” arXiv:2406.13275 , 2024

2024

Closest in time.

H. Dinkel, Y. Wang, Z. Yan, J. Zhang, and Y. Wang, “Ced: Consistent ensemble distillation for audio tagging,” in ICASSP . IEEE, 2024, pp. 291–295

2024

Closest in time.

M. Yang, A. Feng, B. Xiong, J. Liu, I. King, and R. Ying, “Hyperbolic fine-tuning for large language models,” arXiv:2410.04010 , 2024

2024

Closest in time.

J. Zhu, C. Lv, X. Wang, M. Wu, W. Liu, T. Li, Z. Ling, C. Zhang, X. Zheng, and X. Huang, “Promoting data and model privacy in federated learning through quantized lora,” arXiv:2406.10976 , 2024

Original

2024

Closest in time.

J. T. Halloran, M. Gulati, and P. F. Roysdon, “Mamba state-space models can be strong downstream learners,” arXiv:2406.00209 , 2024

2024

Closest in time.

X. Jihong, Z. Xiang, and L. CHENG, “Edge computing for real-time decision making in autonomous driving: Review of challenges, solutions, and future trends.” International Journal of Advanced Computer Science & Applications , vol. 15, no. 7, 2024

2024

Closest in time.

H. Jeon, Y. Kim, and J.-J. Kim, “L4Q: Parameter efficient quantization-aware fine-tuning on large language models,” in ACL , 2025

2025

Closest in time.

Z. Zhou, Q. Zhang, H. Kumbong, and K. Olukotun, “LowRA: Accurate and efficient loRA fine-tuning of LLMs under 2 bits,” in ICML , 2025

2025

Closest in time.

Z. Zhao, T. Shen, D. Zhu, Z. Li, J. Su, X. Wang, and F. Wu, “Merging loRAs like playing LEGO: Pushing the modularity of loRA to extremes through rank-wise clustering,” in ICLR , 2025

2025

Closest in time.

Q. Huang, T. Ko, Z. Zhuang, L. Tang, and Y. Zhang, “HiRA: Parameter-efficient hadamard high-rank adaptation for large language models,” in ICLR , 2025

2025

Closest in time.

P. Albert, F. Z. Zhang, H. Saratchandran, C. Rodriguez-Opazo, A. van den Hengel, and E. Abbasnejad, “RandloRA: Full rank parameter-efficient fine-tuning of large models,” in ICLR , 2025

2025

Closest in time.

Y. Tian, B. Zhang, Z. Tu, and D. Chu, “Adapters selector: Cross-domains and multi-tasks LoRA modules integration usage method,” in COLING , 2025

2025

Closest in time.

Q. Huang, T. Ko, L. Tang, and Y. Zhang, “Comlora: A competitive learning approach for enhancing lora,” in ICLR , 2025

2025

Closest in time.

B. Yu, Z. Yang, and X. Yi, “MoKA:parameter efficiency fine-tuning via mixture of kronecker product adaption,” in COLING , 2025

2025

Closest in time.

T. Truong, C. Nguyen, H. Nguyen, M. Le, T. Le, and N. Ho, “ReploRA: Reparameterizing low-rank adaptation via the perspective of mixture of experts,” in ICML , 2025

2025

Closest in time.

C. Fan, Z. Lu, S. Liu, C. Gu, X. Qu, W. Wei, and Y. Cheng, “Make loRA great again: Boosting loRA with adaptive singular values and mixture-of-experts optimization alignment,” in ICML , 2025

2025

Closest in time.

M. Liao, W. Chen, J. Shen, S. Guo, and H. Wan, “HMoRA: Making LLMs more effective with hierarchical mixture of loRA experts,” in ICLR , 2025

2025

Closest in time.

L. Zhao, W. Zeng, S. Xiaofeng, and H. Zhou, “MoSLD: An extremely parameter-efficient mixture-of-shared LoRAs for multi-task learning,” in COLING , 2025

2025

Closest in time.

Z. Zhao, Y. Zhou, Z. Zhang, D. Zhu, T. Shen, Z. Li, J. Yang, X. Wang, J. Su, K. Kuang, Z. Wei, F. Wu, and Y. Cheng, “Each rank could be an expert: Single-ranked mixture of experts lora for multi-task learning,” 2025

2025

Closest in time.

C. Wang, Y. Lyu, Z. Sun, and L. Jing, “Continual gradient low-rank projection fine-tuning for LLMs,” in ACL , 2025

2025

Closest in time.

Y.-Y. Qian, Y.-Z. Xu, Z.-Y. Zhang, P. Zhao, and Z.-H. Zhou, “TreeloRA: Efficient continual learning via layer-wise loRAs guided by a hierarchical gradient-similarity tree,” in ICML , 2025

2025

Closest in time.

R. Singhal, K. Ponkshe, and P. Vepakomma, “FedEx-LoRA: Exact aggregation for federated and efficient fine-tuning of large language models,” in ACL , 2025

2025

Closest in time.

J. Koo, M. Jang, and J. Ok, “Towards robust and efficient federated low-rank adaptation with heterogeneous clients,” in ACL , 2025

2025

Closest in time.

Y. Zhang, H. Zhu, A. Z. Tan, D. Yu, L. Huang, and H. Yu, “pfedmxf: Personalized federated class-incremental learning with mixture of frequency aggregation,” in Proceedings of the Computer Vision and Pattern Recognition Conference , 2025, pp. 30 640–30 650

2025

Closest in time.

J. Li, Y. Shao, J. Zhuo, C. Li, L. Tang, D. Yu, and Y. Qian, “Efficient multilingual asr finetuning via lora language experts,” arXiv:2506.21555 , 2025

2025

Closest in time.

S. Xia, L. Sun, T. Sun, and Q. Li, “DragloRA: Online optimization of loRA adapters for drag-based image editing in diffusion model,” in ICML , 2025

2025

Closest in time.

S. Zhuang, Y. Guo, Y. Ding, K. Li, X. Chen, Y. Wang, F. Wang, Y. Zhang, C. Li, and Y. Wang, “Timestep master: Asymmetrical mixture of timestep loRA experts for versatile and efficient diffusion models in vision,” in ICML , 2025

2025

Closest in time.

C. Wang, S. Hu, G. Tan, and W. Jia, “ELoRA: Low-rank adaptation for equivariant GNNs,” in ICML , 2025

2025

Closest in time.

J. Zhang, W. Zhu, and J. Gao, “Adapting large language models for time series modeling via a novel parameter-efficient adaptation method,” arXiv:2502.13725 , 2025

2025

Closest in time.

L. Pan, Z. Chen, H. Li, G. Liu, Z. Xu, Z. Liu, H. Wang, and Y. Wei, “Mixture of low rank adaptation with partial parameter sharing for time series forecasting,” arXiv , 2025

2025

Closest in time.

C. Ge, X. Wang, Z. Zhang, H. Chen, J. Fan, L. Huang, H. Xue, and W. Zhu, “Dynamic mixture of curriculum loRA experts for continual multimodal instruction tuning,” in ICML , 2025

2025

Closest in time.