Fetching the paper…
Reading the bibliography…
Large Language Models (LLMs) achieve state-of-the-art performance but are challenging to deploy due to their high computational and storage demands.
Huggingface’s transformers: State-of-the-art natural language processing
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, et al. 2019 · 1910
Earlier work this paper cites.
The penn treebank: Annotating predicate argument structure
Mitch Marcus, Grace Kim, Mary Ann Marcinkiewicz, Robert MacIntyre, Ann Bies, Mark Ferguson, Karen Katz, and Britta Schasberger. 1994 · 1994
Earlier work this paper cites.
Song Han, Huizi Mao, and William J Dally. 2015 · 2015
Earlier work this paper cites.
Pointer sentinel mixture models
Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. 2016 · 2016
Earlier work this paper cites.
Communication-efficient learning of deep networks from decentralized data
Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Arcas. 2017 · 2017
Earlier work this paper cites.
To prune, or not to prune: exploring the efficacy of pruning for model compression
Michael Zhu and Suyog Gupta. 2017 · 2017
Earlier work this paper cites.
Pytorch: An imperative style, high-performance deep learning library
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. 2019 · 2019
Earlier work this paper cites.
Exploring the limits of transfer learning with a unified text-to-text transformer
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020 · 2020
Earlier work this paper cites.
Heterofl: Computation and communication efficient federated learning for heterogeneous clients
Enmao Diao, Jie Ding, and Vahid Tarokh. 2021 · 2021
Earlier work this paper cites.
Accelerated sparse neural training: A provable and efficient method to find n: m transposable masks
Itay Hubara, Brian Chmiel, Moshe Island, Ron Banner, Joseph Naor, and Daniel Soudry. 2021 · 2021
Earlier work this paper cites.
Fedprune: Towards inclusive federated learning
Muhammad Tahir Munir, Muhammad Mustansar Saeed, Mahad Ali, Zafar Ayyub Qazi, and Ihsan Ayyub Qazi. 2021 · 2021
Earlier work this paper cites.
A survey on federated learning
Chen Zhang, Yu Xie, Hang Bai, Bin Yu, Weihong Li, and Yuan Gao. 2021 · 2021
Earlier work this paper cites.
Prunefl: Model pruning enables efficient federated learning on edge devices
Yuang Jiang, Shiqiang Wang, Victor Valls, Bong Jun Ko, Wei-Han Lee, Kin K. Leung, and Leandros Tassiulas. 2022 · 2022
Earlier work this paper cites.
A fast post-training pruning framework for transformers
Woosuk Kwon, Sehoon Kim, Michael W Mahoney, Joseph Hassoun, Kurt Keutzer, and Amir Gholami. 2022 · 2022
Earlier work this paper cites.
Fednlp: Benchmarking federated learning methods for natural language processing tasks
Bill Yuchen Lin, Chaoyang He, Zihang Zeng, Hulin Wang, Yufen Huang, Christophe Dupuy, Rahul Gupta, Mahdi Soltanolkotabi, Xiang Ren, and Salman Avestimehr. 2022 · 2022
Earlier work this paper cites.
nuqmm: Quantized matmul for efficient inference of large-scale generative language models
Gunho Park, Baeseong Park, Se Jung Kwon, Byeongwook Kim, Youngjoo Lee, and Dongsoo Lee. 2022 · 2022
Cited alongside, same era.
Zeroquant: Efficient and affordable post-training quantization for large-scale transformers
Zhewei Yao, Reza Yazdani Aminabadi, Minjia Zhang, Xiaoxia Wu, Conglong Li, and Yuxiong He. 2022 · 2022
Cited alongside, same era.
Opt: Open pre-trained transformer language models
Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. 2022 · 2022
Cited alongside, same era.
Fedadapter: Efficient federated learning for modern nlp
Dongqi Cai, Yaozong Wu, Shangguang Wang, Felix Xiaozhu Lin, and Mengwei Xu. 2023 · 2023
Cited alongside, same era.
The case for 4-bit precision: k-bit inference scaling laws
Tim Dettmers and Luke Zettlemoyer. 2023 · 2023
A simple and effective pruning approach for large language models
Mingjie Sun, Zhuang Liu, Anna Bair, and J Zico Kolter. 2023 · 2023
Later among the works it cites.
Structured pruning for efficient generative pre-trained language models
Chaofan Tao, Lu Hou, Haoli Bai, Jiansheng Wei, Xin Jiang, Qun Liu, Ping Luo, and Ngai Wong. 2023 · 2023
Later among the works it cites.
Llama 2: Open foundation and fine-tuned chat models
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023 · 2023
Later among the works it cites.
Acceltran: A sparsity-aware accelerator for dynamic inference with transformers
Shikhar Tuli and Niraj K Jha. 2023 · 2023
Later among the works it cites.
Dafkd: Domain-aware federated knowledge distillation
Haozhao Wang, Yichen Li, Wenchao Xu, Ruixuan Li, Yufeng Zhan, and Zhigang Zeng. 2023 · 2023
alphaXiv searches the wider corpus for related work and actual follow-ups.
alphaXiv is searching for related work…
Cited alongside, same era.
Massive language models can be accurately pruned in one-shot
Elias Frantar and Dan Alistarh. 2023 · 2023
Cited alongside, same era.
Dynamic activation of clients and parameters for federated learning over heterogeneous graphs
Zishan Gu, Ke Zhang, Guangji Bai, Liang Chen, Liang Zhao, and Carl Yang. 2023 · 2023
Cited alongside, same era.
Distributed pruning towards tiny neural networks in federated learning
Hong Huang, Lan Zhang, Chaoyue Sun, Ruogu Fang, Xiaoyong Yuan, and Dapeng Wu. 2023 · 2023
Cited alongside, same era.
Complement sparsification: Low-overhead model pruning for federated learning
Xiaopeng Jiang and Cristian Borcea. 2023 · 2023
Cited alongside, same era.
Client-customized adaptation for parameter-efficient federated learning
Yeachan Kim, Junho Kim, Wing-Lam Mok, Jun-Hyung Park, and SangKeun Lee. 2023 · 2023
Cited alongside, same era.
Ziplm: Hardware-aware structured pruning of language models
Eldar Kurtic, Elias Frantar, and Dan Alistarh. 2023 · 2023
Cited alongside, same era.
Losparse: Structured compression of large language models based on low-rank and sparse approximation
Yixiao Li, Yifan Yu, Qingru Zhang, Chen Liang, Pengcheng He, Weizhu Chen, and Tuo Zhao. 2023 · 2023
Cited alongside, same era.
Later among the works it cites.
Fedprompt: Communication-efficient and privacy preserving prompt tuning in federated learning
Haodong Zhao, Wei Du, Fangqi Li, Peixuan Li, and Gongshen Liu. 2023 · 2023
Later among the works it cites.
A survey on model compression for large language models
Xunyu Zhu, Jian Li, Yong Liu, Can Ma, and Weiping Wang. 2023 · 2023
Later among the works it cites.
Automated federated pipeline for parameter-efficient fine-tuning of large language models
Zihan Fang, Zheng Lin, Zhe Chen, Xianhao Chen, Yue Gao, and Yuguang Fang. 2024 · 2024
Closest in time.
Pre-text: Training language models on private federated data in the age of llms
Charlie Hou, Akshat Shrivastava, Hongyuan Zhan, Rylan Conway, Trang Le, Adithya Sagar, Giulia Fanti, and Daniel Lazar. 2024 · 2024
Closest in time.
Advances and open challenges in federated learning with foundation models
Chao Ren, Han Yu, Hongyi Peng, Xiaoli Tang, Anran Li, Yulan Gao, Alysa Ziying Tan, Bo Zhao, Xiaoxiao Li, Zengxiang Li, et al. 2024 · 2024
Closest in time.
Fedbpt: Efficient federated black-box prompt tuning for large language models
Jingwei Sun, Ziyue Xu, Hongxu Yin, Dong Yang, Daguang Xu, Yiran Chen, and Holger R. Roth. 2024 · 2024
Closest in time.
Fedbiot: Llm local fine-tuning in federated learning without full model
Feijie Wu, Zitao Li, Yaliang Li, Bolin Ding, and Jing Gao. 2024 · 2024
Closest in time.
Federa: Efficient fine-tuning of language models in federated learning leveraging weight decomposition
Yuxuan Yan, Shunpu Tang, Zhiguo Shi, and Qianqian Yang. 2024 · 2024
Closest in time.
Fedp3: Federated personalized and privacy-friendly network pruning under model heterogeneity
Kai Yi, Nidham Gazagnadou, Peter Richtarik, and Lingjuan Lyu. 2024 · 2024
Closest in time.