SplitLoRA: A Split Parameter-Efficient Fine-Tuning Framework for Large Language Models

K. Simonyan and A. Zisserman, “Very Deep Convolutional Networks for Large-scale Image Recognition,” in Proc. ICLR , 2015

2015

Earlier work this paper cites.

J. Konečnỳ, H. B. McMahan, F. X. Yu, P. Richtárik, A. T. Suresh, and D. Bacon, “Federated Learning: Strategies for Improving Communication Efficiency,” arXiv preprint arXiv:1610.05492 , 2016

Original

2016

Earlier work this paper cites.

K. He, X. Zhang, S. Ren, and J. Sun, “Deep Residual Learning for Image Recognition,” in Proc. CVPR , 2016, pp. 770–778

2016

Earlier work this paper cites.

J. Novikova, O. Dušek, and V. Rieser, “The E2E Dataset: New Challenges For End-to-End Generation,” in Proc. SIGDIAL , 2017

2017

Earlier work this paper cites.

B. McMahan, E. Moore, D. Ramage, S. Hampson, and B. A. y Arcas, “Communication-efficient Learning of Deep Networks From Decentralized Data,” in Proc. AISTATS , 2017

2017

Earlier work this paper cites.

P. Vepakomma, O. Gupta, T. Swedish, and R. Raskar, “Split Learning for Health: Distributed Deep Learning without Sharing Raw Patient Data,” arXiv preprint arXiv:1812.00564 , 2018

Original

2018

Earlier work this paper cites.

J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “Bert: Pre-training of Deep Bidirectional Transformers for Language Understanding,” arXiv preprint arXiv:1810.04805 , 2018

Original

2018

Earlier work this paper cites.

A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, I. Sutskever et al. , “Language Models are Unsupervised Multitask Learners,” OpenAI blog , vol. 1, no. 8, p. 9, 2019

2019

Earlier work this paper cites.

N. Houlsby, A. Giurgiu, S. Jastrzebski, B. Morrone, Q. De Laroussilhe, A. Gesmundo, M. Attariyan, and S. Gelly, “Parameter-Efficient Transfer Learning for NLP,” in Proc. ICML , 2019, pp. 2790–2799

2019

Earlier work this paper cites.

M. Nasr, R. Shokri, and A. Houmansadr, “Comprehensive Privacy Analysis of Deep Learning: Passive and Active White-box Inference Attacks Against Centralized and Federated Learning,” in Proc. SP , 2019

2019

Earlier work this paper cites.

J. Kaplan, S. McCandlish, T. Henighan, T. B. Brown, B. Chess, R. Child, S. Gray, A. Radford, J. Wu, and D. Amodei, “Scaling Laws for Neural Language Models,” arXiv preprint arXiv:2001.08361 , 2020

Original

2020

Earlier work this paper cites.

K. Wei, J. Li, M. Ding, C. Ma, H. H. Yang, F. Farokhi, S. Jin, T. Q. Quek, and H. V. Poor, “Federated Learning with Differential Privacy: Algorithms and Performance Analysis,” IEEE Trans. Inf. Forensics Secur. , vol. 15, pp. 3454–3469, 2020

Original

2020

Earlier work this paper cites.

E. J. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, and W. Chen, “Lora: Low-rank Adaptation of Large Language Models,” arXiv preprint arXiv:2106.09685 , 2021

Original

2021

Earlier work this paper cites.

R. Dale, “GPT-3: What’s It Good For?” Nat. Lang. Eng. , vol. 27, no. 1, pp. 113–118, 2021

2021

Earlier work this paper cites.

Z. Nan, H. Guan, X. Shen, and C. Liao, “Deep NLP-Based Co-evolvement for Synthesizing Code Analysis from Natural Language,” in Proc. of CC , 2021

2021

Earlier work this paper cites.

J. Pfeiffer, A. Kamath, A. Rücklé, K. Cho, and I. Gurevych, “AdapterFusion: Non-Destructive Task Composition for Transfer Learning,” in Proc. EACL , 2021, pp. 487–503

2021

Earlier work this paper cites.

R. Karimi Mahabadi, J. Henderson, and S. Ruder, “Compacter: Efficient Low-Rank Hypercomplex Adapter Layers,” 2021, pp. 1022–1035

2021

Earlier work this paper cites.

X. L. Li and P. Liang, “Prefix-tuning: Optimizing Continuous Prompts for Generation,” arXiv preprint arXiv:2101.00190 , 2021

Original

2021

Earlier work this paper cites.

X. Liu, K. Ji, Y. Fu, W. L. Tam, Z. Du, Z. Yang, and J. Tang, “P-tuning v2: Prompt Tuning Can Be Comparable to Fine-tuning Universally Across Scales and Tasks,” arXiv preprint arXiv:2110.07602 , 2021

Original

2021

Earlier work this paper cites.

A. Aghajanyan, S. Gupta, and L. Zettlemoyer, “Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuning,” in Proc. IJCNLP , 2021, pp. 7319–7328

2021

Earlier work this paper cites.

D. Pasquini, G. Ateniese, and M. Bernaschi, “Unleashing the Tiger: Inference Attacks on Split Learning,” in Proc. CCS , 2021

2021

Earlier work this paper cites.

N. Carlini, F. Tramer, E. Wallace, M. Jagielski, A. Herbert-Voss, K. Lee, A. Roberts, T. Brown, D. Song, U. Erlingsson et al. , “Extracting Training Data From Large Language Models,” in USENIX Security , 2021, pp. 2633–2650

2021

Earlier work this paper cites.

A. Chowdhery, S. Narang, J. Devlin, M. Bosma, G. Mishra, A. Roberts, P. Barham, H. W. Chung, C. Sutton, S. Gehrmann et al. , “Palm: Scaling Language Modeling with Pathways,” arXiv preprint arXiv:2204.02311 , 2022

Original

2022

Earlier work this paper cites.

P. Villalobos, J. Sevilla, L. Heim, T. Besiroglu, M. Hobbhahn, and A. Ho, “Will We Run Out of Data? An Analysis of the Limits of Scaling Datasets in Machine Learning,” arXiv preprint arXiv:2211.04325 , 2022

Original

2022

Earlier work this paper cites.

C. Thapa, P. C. M. Arachchige, S. Camtepe, and L. Sun, “Splitfed: When Federated Learning Meets Split Learning,” in Proc. AAAI , 2022

2022

Earlier work this paper cites.

Z. Lin, L. Wang, J. Ding, B. Tan, and S. Jin, “Channel power gain estimation for terahertz vehicle-to-infrastructure networks,” IEEE Commun. Lett. , vol. 27, no. 1, pp. 155–159, 2022

2022

Earlier work this paper cites.

X. Liu, Y. Deng, and T. Mahmoodi, “Wireless Distributed Learning: A New Hybrid Split and Federated Learning Approach,” IEEE Trans. Wireless Commun. , vol. 22, no. 4, pp. 2650–2665, 2022

2022

Cited alongside, same era.

Huawei, NET4AI: Supporting AI as a Service in 6G . Cambridge, U.K.: Cambridge Univ. Press, 2022

2022

Cited alongside, same era.

S. Gupta, Y. Huang, Z. Zhong, T. Gao, K. Li, and D. Chen, “Recovering Private Text in Federated Learning of Language Models,” Proc. NIPS , 2022

2022

Cited alongside, same era.

OpenAI, “GPT-4 Technical Report,” arXiv preprint arXiv:2303.08774 , 2023

Original

2023

Cited alongside, same era.

H. Touvron, L. Martin, K. Stone, P. Albert, A. Almahairi, Y. Babaei, N. Bashlykov, S. Batra, P. Bhargava, S. Bhosale et al. , “Llama 2: Open Foundation and Fine-tuned Chat Models,” arXiv preprint arXiv:2307.09288 , 2023

Z. Cheng, X. Xia, M. Liwang, X. Fan, Y. Sun, X. Wang, and L. Huang, “CHEESE: Distributed Clustering-based Hybrid Federated Split Learning over Edge Networks,” IEEE Trans. Parallel Distrib. Syst. , 2023

2023

Later among the works it cites.

L. Li, Y. Zhang, and L. Chen, “Prompt Distillation for Efficient LLM-based Recommendation,” in Proc. CIKM , 2023, pp. 1348–1357

2023

Later among the works it cites.

C. Chen, X. Feng, J. Zhou, J. Yin, and X. Zheng, “Federated Large Language Model: A Position Paper,” arXiv preprint arXiv:2307.08925 , 2023

Original

2023

Later among the works it cites.

T. Fan, Y. Kang, G. Ma, W. Chen, W. Wei, L. Fan, and Q. Yang, “Fate-LLM: A Industrial Grade Federated Learning Framework for Large Language Models,” arXiv preprint arXiv:2310.10049 , 2023

Original

2023

Later among the works it cites.

W. Kuang, B. Qian, Z. Li, D. Chen, D. Gao, X. Pan, Y. Xie, Y. Li, B. Ding, and J. Zhou, “FederatedScope-LLM: A Comprehensive Package for Fine-tuning Large Language Models in Federated Learning,” arXiv preprint arXiv:2309.00363 , 2023