Fetching the paper…
Reading the bibliography…
We investigate LoRA in federated learning through the lens of the asymmetry analysis of the learned $A$ and $B$ matrices.
An overview of gradient descent optimization algorithms
Sebastian Ruder · 2016
Earlier work this paper cites.
Decoupled weight decay regularization
Ilya Loshchilov and Frank Hutter · 2017
Earlier work this paper cites.
Communication-efficient learning of deep networks from decentralized data
Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Arcas · 2017
Earlier work this paper cites.
Measuring the intrinsic dimension of objective landscapes
Chunyuan Li, Heerad Farkhoor, Rosanne Liu, and Jason Yosinski · 2018
Earlier work this paper cites.
Glue: A multi-task benchmark and analysis platform for natural language understanding
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman · 2018
Earlier work this paper cites.
Federated learning with non-iid data
Yue Zhao, Meng Li, Liangzhen Lai, Naveen Suda, Damon Civin, and Vikas Chandra · 2018
Earlier work this paper cites.
Qsparse-local-sgd: Distributed sgd with quantization, sparsification and local computations
Debraj Basu, Deepesh Data, Can Karakus, and Suhas Diggavi · 2019
Earlier work this paper cites.
Parameter-efficient transfer learning for nlp
Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly · 2019
Earlier work this paper cites.
Codesearchnet challenge: Evaluating the state of semantic code search
Hamel Husain, Ho-Hsiang Wu, Tiferet Gazit, Miltiadis Allamanis, and Marc Brockschmidt · 2019
Earlier work this paper cites.
Roberta: A robustly optimized bert pretraining approach
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov · 2019
Earlier work this paper cites.
Parallel restarted sgd with faster convergence and less communication: Demystifying why model averaging works for deep learning
Hao Yu, Sen Yang, and Shenghuo Zhu · 2019
Earlier work this paper cites.
Intrinsic dimensionality explains the effectiveness of language model fine-tuning
Armen Aghajanyan, Luke Zettlemoyer, and Sonal Gupta · 2020
Earlier work this paper cites.
Language models are few-shot learners
Tom B Brown · 2020
Earlier work this paper cites.
Fetchsgd: Communication-efficient federated learning with sketching
Daniel Rothchild, Ashwinee Panda, Enayat Ullah, Nikita Ivkin, Ion Stoica, Vladimir Braverman, Joseph Gonzalez, and Raman Arora · 2020
Earlier work this paper cites.
Personalized federated learning with moreau envelopes
Canh T Dinh, Nguyen Tran, and Josh Nguyen · 2020
Earlier work this paper cites.
Transformers: State-of-the-art natural language processing
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, et al · 2020
Earlier work this paper cites.
Training verifiers to solve math word problems
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al · 2021
Earlier work this paper cites.
Exploiting shared representations for personalized federated learning
Liam Collins, Hamed Hassani, Aryan Mokhtari, and Sanjay Shakkottai · 2021
Earlier work this paper cites.
Advances and open problems in federated learning
Peter Kairouz, H Brendan McMahan, Brendan Avent, Aurélien Bellet, Mehdi Bennis, Arjun Nitin Bhagoji, Kallista Bonawitz, Zachary Charles, Graham Cormode, Rachel Cummings, et al · 2021
Earlier work this paper cites.
Prefix-tuning: Optimizing continuous prompts for generation
Xiang Lisa Li and Percy Liang · 2021
Earlier work this paper cites.
A survey on federated learning
Chen Zhang, Yu Xie, Hang Bai, Bin Yu, Weihong Li, and Yuan Gao · 2021
Earlier work this paper cites.
Federated learning on non-iid data: A survey
Hangyu Zhu, Jinjin Xu, Shiqing Liu, and Yaochu Jin · 2021
Cited alongside, same era.
Lora: Low-rank adaptation of large language models
Edward J Hu, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al · 2022
Cited alongside, same era.
Fedscale: Benchmarking model and system performance of federated learning at scale
Fan Lai, Yinwei Dai, Sanjay Singapuram, Jiachen Liu, Xiangfeng Zhu, Harsha Madhyastha, and Mosharaf Chowdhury · 2022
Cited alongside, same era.
Federated learning on non-iid data silos: An experimental study
Qinbin Li, Yiqun Diao, Quan Chen, and Bingsheng He · 2022
Cited alongside, same era.
Rethinking architecture design for tackling data heterogeneity in federated learning
Liangqiong Qu, Yuyin Zhou, Paul Pu Liang, Yingda Xia, Feifei Wang, Ehsan Adeli, Li Fei-Fei, and Daniel Rubin · 2022
Cited alongside, same era.
Pretrained models for multilingual federated learning
Towards federated low-rank adaptation with rank-heterogeneous communication
Yuji Byun and Jaeho Lee · 2024
Closest in time.
Linxiao Cao, Yifei Zhu, and Wei Gong · 2024
Closest in time.
Internal cross-layer gradients for extending homogeneity to heterogeneity in federated learning
Yun-Hin Chan, Rui Zhou, Running Zhao, Zhihan JIANG, and Edith CH Ngai · 2024
Closest in time.
Rbla: Rank-based-lora-aggregation for fine-tuning heterogeneous models in flaas, 2024
Shuaijun Chen, Omid Tavallaie, Niousha Nazemi, and Albert Y. Zomaya · 2024
Closest in time.
Harmonizing generalization and personalization in federated prompt learning
Tianyu Cui, Hongxia Li, Jingya Wang, and Ye Shi · 2024
Closest in time.
alphaXiv searches the wider corpus for related work and actual follow-ups.
alphaXiv is searching for related work…
Orion Weller, Marc Marone, Vladimir Braverman, Dawn Lawrie, and Benjamin Van Durme · 2022
Cited alongside, same era.
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al · 2023
Cited alongside, same era.
Federated learning of large language models with parameter-efficient prompt tuning and adaptive optimization
Tianshi Che, Ji Liu, Yang Zhou, Jiaxiang Ren, Jiwen Zhou, Victor Sheng, Huaiyu Dai, and Dejing Dou · 2023
Cited alongside, same era.
On the importance and applicability of pre-training for federated learning
Hong-You Chen, Cheng-Hao Tu, Ziwei Li, Han Wei Shen, and Wei-Lun Chao · 2023
Cited alongside, same era.
Heterogeneous lora for federated fine-tuning of on-device foundation models
Yae Jee Cho, Luyang Liu, Zheng Xu, Aldi Fahrezi, Matt Barnes, and Gauri Joshi · 2023
Cited alongside, same era.
Palm: Scaling language modeling with pathways
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al · 2023
Cited alongside, same era.
Pfedprompt: Learning personalized prompt for vision-language models in federated learning
Tao Guo, Song Guo, and Junxiao Wang · 2023
Cited alongside, same era.
Unlocking the potential of prompt-tuning in bridging generalized and personalized federated learning
Wenlong Deng, Christos Thrampoulidis, and Xiaoxiao Li · 2024
Closest in time.
Lora+: Efficient low rank adaptation of large models
Soufiane Hayou, Nikhil Ghosh, and Bin Yu · 2024
Closest in time.
Vera: Vector-based random matrix adaptation
Dawid Jan Kopiczko, Tijmen Blankevoort, and Yuki M Asano · 2024
Closest in time.
Global and local prompts cooperation via optimal transport for federated learning
Hongxia Li, Wei Huang, Jingya Wang, and Ye Shi · 2024
Closest in time.
Splitlora: A split parameter-efficient fine-tuning framework for large language models
Zheng Lin, Xuanjie Hu, Yuxin Zhang, Zhe Chen, Zihan Fang, Xianhao Chen, Ang Li, Praneeth Vepakomma, and Yue Gao · 2024
Closest in time.
Dora: Weight-decomposed low-rank adaptation
Shih-yang Liu, Chien-Yi Wang, Hongxu Yin, Pavlo Molchanov, Yu-Chiang Frank Wang, Kwang-Ting Cheng, and Min-Hung Chen · 2024
Closest in time.
Introducing meta llama 3: The most capable openly available llm to date
AI Meta · 2024
Closest in time.
Fdlora: Personalized federated learning of large language model via dual lora tuning
Jiaxing Qi, Zhongzhi Luan, Shaohan Huang, Carol Fung, Hailong Yang, and Depei Qian · 2024
Closest in time.
Federated text-driven prompt generation for vision-language models
Chen Qiu, Xingyu Li, Chaithanya Kumar Mummadi, Madan Ravi Ganesh, Zhenzhen Li, Lu Peng, and Wan-Yi Lin · 2024
Closest in time.
Hydralora: An asymmetric lora architecture for efficient fine-tuning
Chunlin Tian, Zhan Shi, Zhijiang Guo, Li Li, and Chengzhong Xu · 2024
Closest in time.
Fedbiot: Llm local fine-tuning in federated learning without full model
Feijie Wu, Zitao Li, Yaliang Li, Bolin Ding, and Jing Gao · 2024
Closest in time.
Fedconv: Enhancing convolutional neural networks for handling data heterogeneity in federated learning
Peiran Xu, Zeyu Wang, Jieru Mei, Liangqiong Qu, Alan Yuille, Cihang Xie, and Yuyin Zhou · 2024
Closest in time.
Dual-personalizing adapter for federated foundation models
Yiyuan Yang, Guodong Long, Tao Shen, Jing Jiang, and Michael Blumenstein · 2024
Closest in time.
Tackling data heterogeneity in federated learning via loss decomposition
Shuang Zeng, Pengxin Guo, Shuai Wang, Jianbo Wang, Yuyin Zhou, and Liangqiong Qu · 2024
Closest in time.
Asymmetry in low-rank adapters of foundation models
Jiacheng Zhu, Kristjan Greenewald, Kimia Nadjahi, Haitz Sáez de Ocáriz Borde, Rickard Brüel Gabrielsson, Leshem Choshen, Marzyeh Ghassemi, Mikhail Yurochkin, and Justin Solomon · 2024
Closest in time.