Arcee's MergeKit: A Toolkit for Merging Large Language Models

Huggingface’s transformers: State-of-the-art natural language processing

Original

Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, et al. 2019 · 1910

Earlier work this paper cites.

Animating rotation with quaternion curves

Ken Shoemake. 1985 · 1985

Earlier work this paper cites.

Weight averaging for neural networks and local resampling schemes

Joachim Utans. 1996 · 1996

Earlier work this paper cites.

What is being transferred in transfer learning?

Original

Behnam Neyshabur, Hanie Sedghi, and Chiyuan Zhang. 2021 · 2008

Earlier work this paper cites.

An investigation of how neural networks learn from the experiences of peers through periodic weight averaging

Joshua Smith and Michael Gashler. 2017 · 2017

Earlier work this paper cites.

Loss surfaces, mode connectivity, and fast ensembling of dnns

Timur Garipov, Pavel Izmailov, Dmitrii Podoprikhin, Dmitry P Vetrov, and Andrew G Wilson. 2018 · 2018

Earlier work this paper cites.

Averaging weights leads to wider optima and better generalization

Original

Pavel Izmailov, Dmitrii Podoprikhin, Timur Garipov, Dmitry Vetrov, and Andrew Gordon Wilson. 2018 · 2018

Earlier work this paper cites.

PubMedQA: A dataset for biomedical research question answering

Qiao Jin, Bhuwan Dhingra, Zhengping Liu, William Cohen, and Xinghua Lu. 2019 · 2019

Earlier work this paper cites.

Uniform convergence may be unable to explain generalization in deep learning

Vaishnavh Nagarajan and J Zico Kolter. 2019 · 2019

Earlier work this paper cites.

Model fusion via optimal transport

Sidak Pal Singh and Martin Jaggi. 2020 · 2020

Earlier work this paper cites.

Optimizing mode connectivity via neuron alignment

Norman Tatro, Pin-Yu Chen, Payel Das, Igor Melnyk, Prasanna Sattigeri, and Rongjie Lai. 2020 · 2020

Earlier work this paper cites.

A continual learning survey: Defying forgetting in classification tasks

Matthias De Lange, Rahaf Aljundi, Marc Masana, Sarah Parisot, Xu Jia, Aleš Leonardis, Gregory Slabaugh, and Tinne Tuytelaars. 2021 · 2021

Earlier work this paper cites.

The role of permutation invariance in linear mode connectivity of neural networks

Original

Rahim Entezari, Hanie Sedghi, Olga Saukh, and Behnam Neyshabur. 2021 · 2021

Earlier work this paper cites.

What disease does this patient have? a large-scale open domain question answering dataset from medical exams

Di Jin, Eileen Pan, Nassim Oufattole, Wei-Hung Weng, Hanyi Fang, and Peter Szolovits. 2021 · 2021

Earlier work this paper cites.

Multitask prompted training enables zero-shot task generalization

Original

Victor Sanh, Albert Webson, Colin Raffel, Stephen H Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Teven Le Scao, Arun Raja, et al. 2021 · 2021

Earlier work this paper cites.

Git re-basin: Merging models modulo permutation symmetries

Original

Samuel K Ainsworth, Jonathan Hayase, and Siddhartha Srinivasa. 2022 · 2022

Earlier work this paper cites.

Editing models with task arithmetic

Original

Gabriel Ilharco, Marco Tulio Ribeiro, Mitchell Wortsman, Suchin Gururangan, Ludwig Schmidt, Hannaneh Hajishirzi, and Ali Farhadi. 2022 · 2022

Earlier work this paper cites.

Dataless knowledge fusion by merging weights of language models

Original

Xisen Jin, Xiang Ren, Daniel Preotiuc-Pietro, and Pengxiang Cheng. 2022 · 2022

Earlier work this paper cites.

Repair: Renormalizing permuted activations for interpolation repair

Original

Keller Jordan, Hanie Sedghi, Olga Saukh, Rahim Entezari, and Behnam Neyshabur. 2022 · 2022

Cited alongside, same era.

Merging models with fisher-weighted averaging

Michael S Matena and Colin A Raffel. 2022 · 2022

Cited alongside, same era.

Medmcqa: A large-scale multi-subject multi-choice dataset for medical domain question answering

Ankit Pal, Logesh Kumar Umapathi, and Malaikannan Sankarasubbu. 2022 · 2022

Cited alongside, same era.

Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time

Mitchell Wortsman, Gabriel Ilharco, Samir Ya Gadre, Rebecca Roelofs, Raphael Gontijo-Lopes, Ari S Morcos, Hongseok Namkoong, Ali Farhadi, Yair Carmon, Simon Kornblith, et al. 2022 · 2022

Cited alongside, same era.

Meditron-70b: Scaling medical pretraining for large language models

Original

Zeming Chen, Alejandro Hernández Cano, Angelika Romanou, Antoine Bonnet, Kyle Matoba, Francesco Salvi, Matteo Pagliardini, Simin Fan, Andreas Köpf, Amirkeivan Mohtashami, et al. 2023 · 2023

Language models are super mario: Absorbing abilities from homologous models as a free lunch

Original

Le Yu, Bowen Yu, Haiyang Yu, Fei Huang, and Yongbin Li. 2023 · 2023

Later among the works it cites.

Evolutionary optimization of model merging recipes

Original

Takuya Akiba, Makoto Shing, Yujin Tang, Qi Sun, and David Ha. 2024 · 2024

Closest in time.

Mitigating catastrophic forgetting in language transfer via model merging

Original

Anton Alexandrov, Veselin Raychev, Mark Niklas Müller, Ce Zhang, Martin Vechev, and Kristina Toutanova. 2024 · 2024

Closest in time.

Llm augmented llms: Expanding capabilities through composition

Original

Rachit Bansal, Bidisha Samanta, Siddharth Dalmia, Nitish Gupta, Shikhar Vashishth, Sriram Ganapathy, Abhishek Bapna, Prateek Jain, and Partha Talukdar. 2024 · 2024

Closest in time.