Towards Federated Low-Rank Adaptation of Language Models with Rank Heterogeneity

DistilBERT, a distilled version of BERT: Smaller, faster, cheaper and lighter

Original

Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. 2019 · 1910

Earlier work this paper cites.

DBpedia: A nucleus for a web of open data

Sören Auer, Christian Bizer, Georgi Kobilarov, Jens Lehmann, Richard Cyganiak, and Zachary Ives. 2007 · 2007

Earlier work this paper cites.

Character-level convolutional networks for text classification

Xiang Zhang, Junbo Zhao, and Yann LeCun. 2015 · 2015

Earlier work this paper cites.

Communication-efficient learning of deep networks from decentralized data

Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Arcas. 2017 · 2017

Earlier work this paper cites.

Federated learning with non-iid data

Original

Yue Zhao, Meng Li, Liangzhen Lai, Naveen Suda, Damon Civin, and Vikas Chandra. 2018 · 2018

Earlier work this paper cites.

Parameter-efficient transfer learning for NLP

Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. 2019 · 2019

Earlier work this paper cites.

ALBERT: A lite BERT for self-supervised learning of language representations

Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. 2020 · 2020

Cited alongside, same era.

FedBN: Federated learning on non-iid features via local batch normalization

Xiaoxiao Li, Meirui Jiang, Xiaofei Zhang, Michael Kamp, and Qi Dou. 2021 · 2021

Cited alongside, same era.

FedNLP: Benchmarking federated learning methods for natural language processing tasks

Bill Yuchen Lin, Chaoyang He, Zihang Zeng, Hulin Wang, Yufen Huang, Christophe Dupuy, Rahul Gupta, Mahdi Soltanolkotabi, Xiang Ren, and Salman Avestimehr. 2021 · 2021

Cited alongside, same era.

Federated learning on non-iid data: A survey

Hangyu Zhu, Jinjin Xu, Shiqing Liu, and Yaochu Jin. 2021 · 2021

Cited alongside, same era.

LoRA: Low-rank adaptation of large language models

Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2022 · 2022

Cited alongside, same era.

SLoRA: Federated parameter efficient fine-tuning of language models

Sara Babakniya, Ahmed Roushdy Elkordy, Yahya H Ezzeldin, Qingfeng Liu, Kee-Bong Song, Mostafa El-Khamy, and Salman Avestimehr. 2023 · 2023

Later among the works it cites.

Client-customized adaptation for parameter-efficient federated learning

Yeachan Kim, Junho Kim, Wing-Lam Mok, Jun-Hyung Park, and SangKeun Lee. 2023 · 2023

Later among the works it cites.

LLaMA: Open and efficient foundation language models

Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023 · 2023

Later among the works it cites.

Heterogeneous LoRA for federated fine-tuning of on-device foundation models

Yae Jee Cho, Luyang Liu, Zheng Xu, Aldi Fahrezi, Matt Barnes, and Gauri Joshi. 2024 · 2024

Closest in time.

Improving LoRA in privacy-preserving federated learning

Youbang Sun, Zitao Li, Yaliang Li, and Bolin Ding. 2024 · 2024

Closest in time.