Fetching the paper…
Reading the bibliography…
Federated Learning enables the fine-tuning of foundation models (FMs) across distributed clients for specific tasks; however, its scalability is limited by the heterogeneity of client memory capacities.
Layer-wise Importance Matters: Less Memory for Better Performance in Parameter-efficient Fine-tuning of Large Language Models. In Findings of the Association for Computational Linguistics: EMNLP 2024 . 1977–1992
Kai Yao, Penglei Gao, Lichun Li, Yuan Zhao, Xiaofeng Wang, Wei Wang, and Jianke Zhu. 2024 · 1992
Earlier work this paper cites.
Information theory and statistics
Solomon Kullback. 1997 · 1997
Earlier work this paper cites.
Learning multiple layers of features from tiny images. In Toronto, ON, Canada
Alex Krizhevsky, Geoffrey Hinton, et al · 2009
Earlier work this paper cites.
Adam: A method for stochastic optimization
Diederik P Kingma. 2014 · 2014
Earlier work this paper cites.
Communication-efficient learning of deep networks from decentralized data. In Artificial intelligence and statistics . PMLR
Brendan McMahan, Eider Moore, Daniel Ramage, et al · 2017
Earlier work this paper cites.
Entropy and mutual information in models of deep neural networks
Marylou Gabrié, Andre Manoel, Clément Luneau, Nicolas Macris, Florent Krzakala, Lenka Zdeborová, et al · 2018
Earlier work this paper cites.
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In Proceedings of the 2019 Conference of the NAACL: Human Language Technologies, Volume 1
Jacob Devlin, Ming-Wei Chang, Kenton Lee, et al · 2019
Earlier work this paper cites.
Moment matching for multi-source domain adaptation. In Proceedings of the ICCV
Xingchao Peng, Qinxun Bai, Xide Xia, et al · 2019
Earlier work this paper cites.
An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale. In ICLR
Alexey Dosovitskiy, Lucas Beyer, et al · 2020
Earlier work this paper cites.
Deepspeed: System optimizations enable training deep learning models with over 100 billion parameters. In Proceedings of the 26th ACM SIGKDD international conference on knowledge discovery & data mining . 3505–3506
Jeff Rasley, Samyam Rajbhandari, Olatunji Ruwase, and Yuxiong He. 2020 · 2020
Earlier work this paper cites.
LoRA: Low-Rank Adaptation of Large Language Models. In ICLR
Edward J Hu, Phillip Wallis, Zeyuan Allen-Zhu, et al · 2021
Earlier work this paper cites.
The Power of Scale for Parameter-Efficient Prompt Tuning. In Proceedings of the 2021 Conference on EMNLP . 3045–3059
Brian Lester, Rami Al-Rfou, and Noah Constant. 2021 · 2021
Earlier work this paper cites.
Data-free knowledge distillation for heterogeneous federated learning. In International conference on machine learning . PMLR, 12878–12889
Zhuangdi Zhu, Junyuan Hong, and Jiayu Zhou. 2021 · 2021
Earlier work this paper cites.
LexGLUE: A Benchmark Dataset for Legal Language Understanding in English. In Proceedings of the 60th Annual Meeting of the ACL . Dubln, Ireland
Ilias Chalkidis, Abhik Jana, Dirk Hartung, et al · 2022
Cited alongside, same era.
Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Tasks. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing . 5085–5109
Yizhong Wang, Swaroop Mishra, Pegah Alipoormolabashi, Yeganeh Kordi, Amirreza Mirzaei, Atharva Naik, Arjun Ashok, Arut Selvan Dhanasekaran, Anjana Arunkumar, David Stap, et al · 2022
Cited alongside, same era.
Opt: Open pre-trained transformer language models
Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al · 2022
Cited alongside, same era.
SLoRA: Federated Parameter Efficient Fine-Tuning of Language Models. In International Workshop in Conjunction with NeurIPS 2023
Sara Babakniya, Ahmed Roushdy Elkordy, Yahya H Ezzeldin, et al · 2023
Cited alongside, same era.
Federated fine-tuning of large language models under heterogeneous tasks and client resources. In The Thirty-eighth Annual Conference on Neural Information Processing Systems
Jiamu Bai, Daoyuan Chen, Bingchen Qian, Liuyi Yao, and Yaliang Li. 2024 · 2024
Closest in time.
Data-juicer: A one-stop data processing system for large language models. In Companion of the 2024 International Conference on Management of Data . 120–134
Daoyuan Chen, Yilun Huang, Zhijian Ma, Hesen Chen, Xuchen Pan, Ce Ge, Dawei Gao, Yuexiang Xie, Zhaoyang Liu, Jinyang Gao, et al · 2024
Closest in time.
Heterogeneous lora for federated fine-tuning of on-device foundation models. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing . 12903–12913
Yae Jee Cho, Luyang Liu, Zheng Xu, Aldi Fahrezi, and Gauri Joshi. 2024 · 2024
Closest in time.
Selective Aggregation for Low-Rank Adaptation in Federated Learning
Pengxin Guo, Shuang Zeng, Yanran Wang, Huijie Fan, Feifei Wang, and Liangqiong Qu. 2024 · 2024
Closest in time.
alphaXiv searches the wider corpus for related work and actual follow-ups.
alphaXiv is searching for related work…
Free dolly: Introducing the world’s first truly open instruction-tuned llm
Mike Conover, Matt Hayes, Ankit Mathur, Jianwei Xie, Jun Wan, Sam Shah, Ali Ghodsi, Patrick Wendell, Matei Zaharia, and Reynold Xin. 2023 · 2023
Cited alongside, same era.
Offloading using traditional optimization and machine learning in federated cloud–edge–fog systems: A survey
Binayak Kar, Widhi Yahya, Ying-Dar Lin, and Asad Ali. 2023 · 2023
Cited alongside, same era.
Shangchao Su, Bin Li, and Xiangyang Xue. 2023 · 2023
Cited alongside, same era.
Stanford alpaca: An instruction-following llama model
Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B Hashimoto. 2023 · 2023
Cited alongside, same era.
Llama: Open and efficient foundation language models
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al · 2023
Cited alongside, same era.
Towards Building the FederatedGPT: Federated Instruction Tuning. In International Workshop in Conjunction with NeurIPS 2023
Jianyi Zhang, Saeed Vahidian, Martin Kuo, et al · 2023
Cited alongside, same era.
Lift: Efficient layer-wise fine-tuning for large model models
Ligeng Zhu, Lanxiang Hu, Ji Lin, and Song Han. 2023 · 2023
Cited alongside, same era.
SlimFit: Memory-Efficient Fine-Tuning of Transformer-based Models Using Training Dynamics. In Proceedings of the 2024 Conference of the NAACL: Human Language Technologies . 6218–6236
Arash Ardakani, Altan Haan, Shangyin Tan, et al · 2024
Cited alongside, same era.
Fisher Information-based Efficient Curriculum Federated Learning with Large Language Models. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing . 10497–10523
Ji Liu, Jiaxiang Ren, Ruoming Jin, Zijie Zhang, Yang Zhou, Patrick Valduriez, and Dejing Dou. 2024 · 2024
Closest in time.
LISA: layerwise importance sampling for memory-efficient large language model fine-tuning
Rui Pan, Xiang Liu, Shizhe Diao, Renjie Pi, Jipeng Zhang, Chi Han, and Tong Zhang. 2024 · 2024
Closest in time.
Improving LoRA in Privacy-preserving Federated Learning. In The Twelfth International Conference on Learning Representations
Youbang Sun, Zitao Li, Yaliang Li, and Bolin Ding. 2024 · 2024
Closest in time.
FLoRA: Federated Fine-Tuning Large Language Models with Heterogeneous Low-Rank Adaptations. In The Thirty-eighth Annual Conference on Neural Information Processing Systems
Ziyao Wang, Zheyu Shen, Yexiao He, Guoheng Sun, Hongyi Wang, Lingjuan Lyu, and Ang Li. 2024 · 2024
Closest in time.
FedFMSL: Federated Learning of Foundations Models With Sparsely Activated LoRA
Panlong Wu, Kangshuo Li, Ting Wang, Yanjie Dong, Victor CM Leung, and Fangxin Wang. 2024 · 2024
Closest in time.
FwdLLM: Efficient federated finetuning of large language models with perturbed inferences. In USENIX ATC
Mengwei Xu, Dongqi Cai, Yaozong Wu, Xiang Li, and Shangguang Wang. 2024 · 2024
Closest in time.
FlowerTune: A Cross-Domain Benchmark for Federated Fine-Tuning of Large Language Models
Yan Gao, Massimo Roberto Scamarcia, Javier Fernandez-Marques, Mohammad Naseri, Chong Shen Ng, Dimitris Stripelis, Zexi Li, Tao Shen, Jiamu Bai, Daoyuan Chen, et al · 2025
Closest in time.
Fed-HeLLo: Efficient Federated Foundation Model Fine-Tuning with Heterogeneous LoRA Allocation
Zikai Zhang, Ping Liu, Jiahao Xu, and Rui Hu. 2025 · 2025
Closest in time.