DELLA-Merging: Reducing Interference in Model Merging through Magnitude-Based Sampling

On the mathematical foundations of theoretical statistics

Ronald A Fisher. 1922 · 1922

Earlier work this paper cites.

Fine-tuning pretrained language models: Weight initializations, data orders, and early stopping

Original

Jesse Dodge, Gabriel Ilharco, Roy Schwartz, Ali Farhadi, Hannaneh Hajishirzi, and Noah Smith. 2020 · 2002

Earlier work this paper cites.

To prune, or not to prune: exploring the efficacy of pruning for model compression

Original

Michael Zhu and Suyog Gupta. 2017 · 2017

Earlier work this paper cites.

Optimization based layer-wise magnitude-based pruning for dnn compression

Guiying Li, Chao Qian, Chunhui Jiang, Xiaofen Lu, and Ke Tang. 2018 · 2018

Earlier work this paper cites.

Rethinking the value of network pruning

Original

Zhuang Liu, Mingjie Sun, Tinghui Zhou, Gao Huang, and Trevor Darrell. 2018 · 2018

Earlier work this paper cites.

Improving language understanding by generative pre-training

Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, et al. 2018 · 2018

Earlier work this paper cites.

The lottery ticket hypothesis: Finding sparse, trainable neural networks

Jonathan Frankle and Michael Carbin. 2019 · 2019

Earlier work this paper cites.

Parameter-efficient transfer learning for nlp

Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. 2019 · 2019

Earlier work this paper cites.

Bert: Pre-training of deep bidirectional transformers for language understanding

Jacob Devlin Ming-Wei Chang Kenton and Lee Kristina Toutanova. 2019 · 2019

Earlier work this paper cites.

Exploring the limits of transfer learning with a unified text-to-text transformer

Colin Raffel, Noam M. Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2019 · 2019

Earlier work this paper cites.

Program synthesis with large language models

Original

Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, et al. 2021 · 2021

Earlier work this paper cites.

Training verifiers to solve math word problems

Original

Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. 2021 · 2021

Earlier work this paper cites.

Lora: Low-rank adaptation of large language models

Original

Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2021 · 2021

Earlier work this paper cites.

Layer-adaptive sparsity for the magnitude-based pruning

Jaeho Lee, Sejun Park, Sangwoo Mo, Sungsoo Ahn, and Jinwoo Shin. 2021 · 2021

Cited alongside, same era.

The power of scale for parameter-efficient prompt tuning

Original

Brian Lester, Rami Al-Rfou, and Noah Constant. 2021 · 2021

Cited alongside, same era.

Prefix-tuning: Optimizing continuous prompts for generation

Original

Xiang Lisa Li and Percy Liang. 2021 · 2021

Cited alongside, same era.

Merging models with fisher-weighted averaging

Michael S Matena and Colin A Raffel. 2022 · 2022

Cited alongside, same era.

Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time

Mitchell Wortsman, Gabriel Ilharco, Samir Ya Gadre, Rebecca Roelofs, Raphael Gontijo-Lopes, Ari S Morcos, Hongseok Namkoong, Ali Farhadi, Yair Carmon, Simon Kornblith, et al. 2022 · 2022

Cited alongside, same era.

Code llama: Open foundation models for code

Baptiste Rozière, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Tal Remez, Jérémy Rapin, Artyom Kozhevnikov, Ivan Evtimov, Joanna Bitton, Manish Bhatt, Cristian Canton Ferrer, Aaron Grattafiori, Wenhan Xiong, Alexandre Défossez, Jade Copet, Faisal Azhar, Hugo Touvron, Louis Martin, Nicolas Usunier, Thomas Scialom, and Gabriel Synnaeve. 2023 · 2023

Later among the works it cites.

Outlier weighed layerwise sparsity (owl): A missing secret sauce for pruning llms to high sparsity

Original

Lu Yin, You Wu, Zhenyu Zhang, Cheng-Yu Hsieh, Yaqing Wang, Yiling Jia, Mykola Pechenizkiy, Yi Liang, Zhangyang Wang, and Shiwei Liu. 2023 · 2023

Later among the works it cites.

Language models are super mario: Absorbing abilities from homologous models as a free lunch

Original

Le Yu, Bowen Yu, Haiyang Yu, Fei Huang, and Yongbin Li. 2023 · 2023

Later among the works it cites.

A survey of large language models

Original

Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, et al. 2023 · 2023

Later among the works it cites.