Merge, Ensemble, and Cooperate! A Survey on Collaborative Strategies in the Era of Large Language Models

Language models are few-shot learners

Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020 · 1901

Earlier work this paper cites.

The curious case of neural text degeneration

Original

Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. 2020b · 1904

Earlier work this paper cites.

Rethinking exposure bias in adversarial language modeling

Original

Yifan Xu, Kening Zhang, Haoyu Dong, Yuezhou Sun, Wenlong Zhao, and Tu Zhuowen. 2020 · 1910

Earlier work this paper cites.

Adaptive mixtures of local experts

Robert A Jacobs, Michael I Jordan, Steven J Nowlan, and Geoffrey E Hinton. 1991 · 1991

Earlier work this paper cites.

Stacked generalization

David H Wolpert. 1992 · 1992

Earlier work this paper cites.

Bagging predictors

Leo Breiman. 1996 · 1996

Earlier work this paper cites.

A decision-theoretic generalization of on-line learning and an application to boosting

Yoav Freund and Robert E Schapire. 1997 · 1997

Earlier work this paper cites.

Ensemble methods in machine learning

Thomas G Dietterich. 2000 · 2000

Earlier work this paper cites.

Scaling laws for neural language models

Original

Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. 2020 · 2001

Earlier work this paper cites.

Scaling large learning problems with hard parallel mixtures

Ronan Collobert, Yoshua Bengio, and Samy Bengio. 2003 · 2003

Earlier work this paper cites.

Cooperative multi-agent learning: The state of the art

Liviu Panait and Sean Luke. 2005 · 2005

Earlier work this paper cites.

Learning factored representations in a deep mixture of experts

Original

David Eigen, Marc’Aurelio Ranzato, and Ilya Sutskever. 2013 · 2013

Earlier work this paper cites.

Distilling the knowledge in a neural network

Original

Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. 2015 · 2015

Earlier work this paper cites.

Sequence level training with recurrent neural networks

Original

Marc’Aurelio Ranzato, Sumit Chopra, Michael Auli, and Wojciech Zaremba. 2015 · 2015

Earlier work this paper cites.

Sequence level training with recurrent neural networks

Marc’Aurelio Ranzato, Sumit Chopra, Michael Auli, and Wojciech Zaremba. 2016 · 2016

Earlier work this paper cites.

Trainable greedy decoding for neural machine translation

Original

Jiatao Gu, Kyunghyun Cho, and Victor O. K. Li. 2017 · 2017

Earlier work this paper cites.

Routing networks: Adaptive selection of non-linear functions for multi-task learning

Original

Clemens Rosenbaum, Tim Klinger, and Matthew Riemer. 2017 · 2017

Earlier work this paper cites.

Attention is all you need

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017 · 2017

Earlier work this paper cites.

Loss surfaces, mode connectivity, and fast ensembling of dnns

Original

Timur Garipov, Pavel Izmailov, Dmitrii Podoprikhin, Dmitry Vetrov, and Andrew Gordon Wilson. 2018 · 2018

Earlier work this paper cites.

Averaging weights leads to wider optima and better generalization

Original

Pavel Izmailov, Dmitrii Podoprikhin, Timur Garipov, Dmitry Vetrov, and Andrew Gordon Wilson. 2018 · 2018

Earlier work this paper cites.

Improving language understanding by generative pre-training

Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, et al. 2018 · 2018

Earlier work this paper cites.

Ensemble learning: A survey

Omer Sagi and Lior Rokach. 2018 · 2018

Earlier work this paper cites.

Blockwise parallel decoding for deep autoregressive models

Mitchell Stern, Noam Shazeer, and Jakob Uszkoreit. 2018 · 2018

Earlier work this paper cites.

BERT: Pre-training of deep bidirectional transformers for language understanding

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019 · 2019

Earlier work this paper cites.

Ranking generated summaries by correctness: An interesting but challenging application for natural language inference

Tobias Falke, Leonardo F. R. Ribeiro, Prasetya Ajie Utama, Ido Dagan, and Iryna Gurevych. 2019 · 2019

Earlier work this paper cites.

Assessing the factual accuracy of generated text

Ben Goodrich, Vinay Rao, Mohammad Saleh, and Peter J. Liu. 2019 · 2019

Earlier work this paper cites.

BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension

Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020 · 2020

Earlier work this paper cites.

Federated learning: Challenges, methods, and future directions

Tian Li, Anit Kumar Sahu, Ameet Talwalkar, and Virginia Smith. 2020 · 2020

Earlier work this paper cites.

Exploring the limits of transfer learning with a unified text-to-text transformer

Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020 · 2020

Earlier work this paper cites.

Bleurt: Learning robust metrics for text generation

Thibault Sellam, Dipanjan Das, and Ankur Parikh. 2020 · 2020

Earlier work this paper cites.

Model fusion via optimal transport

Sidak Pal Singh and Martin Jaggi. 2020 · 2020

Earlier work this paper cites.

Bertscore: Evaluating text generation with bert

Tianyi Zhang*, Varsha Kishore*, Felix Wu*, Kilian Q. Weinberger, and Yoav Artzi. 2020 · 2020

Earlier work this paper cites.

GeDi: Generative discriminator guided sequence generation

Ben Krause, Akhilesh Deepak Gotmare, Bryan McCann, Nitish Shirish Keskar, Shafiq Joty, Richard Socher, and Nazneen Fatema Rajani. 2021 · 2021

Earlier work this paper cites.

Dexperts: Decoding-time controlled text generation with experts and anti-experts

Alisa Liu, Maarten Sap, Ximing Lu, Swabha Swayamdipta, Chandra Bhagavatula, Noah A Smith, and Yejin Choi. 2021 · 2021

Earlier work this paper cites.

Looking beyond sentence-level natural language inference for question answering and text summarization

Anshuman Mishra, Dhruvesh Patel, Aparna Vijayakumar, Xiang Lorraine Li, Pavan Kapanipathi, and Kartik Talamadupula. 2021 · 2021

Earlier work this paper cites.

Entity-level factual consistency of abstractive text summarization

Feng Nan, Ramesh Nallapati, Zhiguo Wang, Cícero Nogueira dos Santos, Henghui Zhu, Dejiao Zhang, Kathleen McKeown, and Bing Xiang. 2021 · 2021

Earlier work this paper cites.

Multitask prompted training enables zero-shot task generalization

Original

Victor Sanh, Albert Webson, Colin Raffel, Stephen H Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Teven Le Scao, Arun Raja, et al. 2021 · 2021

Earlier work this paper cites.

FUDGE: Controlled text generation with future discriminators

Kevin Yang and Dan Klein. 2021 · 2021

Earlier work this paper cites.

Bartscore: Evaluating generated text as text generation

Weizhe Yuan, Graham Neubig, and Pengfei Liu. 2021 · 2021

Earlier work this paper cites.

A survey on in-context learning

Original

Qingxiu Dong, Lei Li, Damai Dai, Ce Zheng, Zhiyong Wu, Baobao Chang, Xu Sun, Jingjing Xu, and Zhifang Sui. 2022 · 2022

Earlier work this paper cites.

QAFactEval: Improved QA-based factual consistency evaluation for summarization

Alexander Fabbri, Chien-Sheng Wu, Wenhao Liu, and Caiming Xiong. 2022 · 2022

Earlier work this paper cites.

Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity

William Fedus, Barret Zoph, and Noam Shazeer. 2022 · 2022

Earlier work this paper cites.

Attributed text generation via post-hoc research and revision

Original

Luyu Gao, Zhuyun Dai, Panupong Pasupat, Anthony Chen, Arun Tejasvi Chaganty, Yicheng Fan, Vincent Zhao, N. Lao, Hongrae Lee, Da-Cheng Juan, and Kelvin Guu. 2022 · 2022

Earlier work this paper cites.

Promptfl: Let federated participants cooperatively learn prompts instead of models – federated learning in age of foundation model

Original

Tao Guo, Song Guo, Junxiao Wang, and Wenchao Xu. 2022 · 2022

Earlier work this paper cites.

Training compute-optimal large language models

Original

Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. 2022 · 2022

Earlier work this paper cites.

LoRA: Low-rank adaptation of large language models

Edward J Hu, yelong shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2022 · 2022

Earlier work this paper cites.

Unsupervised dense information retrieval with contrastive learning

Gautier Izacard, Mathilde Caron, Lucas Hosseini, Sebastian Riedel, Piotr Bojanowski, Armand Joulin, and Edouard Grave. 2021 · 2022

Earlier work this paper cites.

Few-shot learning with retrieval augmented language models

Original

Gautier Izacard, Patrick Lewis, Maria Lomeli, Lucas Hosseini, Fabio Petroni, Timo Schick, Jane A. Yu, Armand Joulin, Sebastian Riedel, and Edouard Grave. 2022 · 2022

Earlier work this paper cites.

Dataless knowledge fusion by merging weights of language models

Original

Xisen Jin, Xiang Ren, Daniel Preotiuc-Pietro, and Pengxiang Cheng. 2022 · 2022

Earlier work this paper cites.

Branch-train-merge: Embarrassingly parallel training of expert language models

Original

Margaret Li, Suchin Gururangan, Tim Dettmers, Mike Lewis, Tim Althoff, Noah A Smith, and Luke Zettlemoyer. 2022 · 2022

Earlier work this paper cites.

Few-shot learning with multilingual generative language models

Xi Victoria Lin, Todor Mihaylov, Mikel Artetxe, Tianlu Wang, Shuohui Chen, Daniel Simig, Myle Ott, Naman Goyal, Shruti Bhosale, Jingfei Du, Ramakanth Pasunuru, Sam Shleifer, Punit Singh Koura, Vishrav Chaudhary, Brian O’Horo, Jeff Wang, Luke Zettlemoyer, Zornitsa Kozareva, Mona Diab, Veselin Stoyanov, and Xian Li. 2022 · 2022

Earlier work this paper cites.

Merging models with fisher-weighted averaging

Michael S Matena and Colin A Raffel. 2022 · 2022

Earlier work this paper cites.

Varifocal question generation for fact-checking

Nedjma Djouhra Ousidhoum, Moy Yuan, and Andreas Vlachos. 2022 · 2022

Earlier work this paper cites.

Training language models to follow instructions with human feedback

Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul F Christiano, Jan Leike, and Ryan Lowe. 2022 · 2022

Earlier work this paper cites.

Diverse weight averaging for out-of-distribution generalization

Alexandre Rame, Matthieu Kirchmeyer, Thibaud Rahier, Alain Rakotomamonjy, Patrick Gallinari, and Matthieu Cord. 2022 · 2022

Cited alongside, same era.

Lamda: Language models for dialog applications

Original

Romal Thoppilan, Daniel De Freitas, Jamie Hall, Noam Shazeer, Apoorv Kulshreshtha, Heng-Tze Cheng, Alicia Jin, Taylor Bos, Leslie Baker, Yu Du, et al. 2022 · 2022

Cited alongside, same era.

Rationale-augmented ensembles in language models

Original

Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, and Denny Zhou. 2022 · 2022

Cited alongside, same era.

Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time

Mitchell Wortsman, Gabriel Ilharco, Samir Ya Gadre, Rebecca Roelofs, Raphael Gontijo-Lopes, Ari S Morcos, Hongseok Namkoong, Ali Farhadi, Yair Carmon, Simon Kornblith, et al. 2022 · 2022

Cited alongside, same era.

End-to-end multimodal fact-checking and explanation generation: A challenging dataset and models

Language models are super mario: Absorbing abilities from homologous models as a free lunch

Original

Le Yu, Bowen Yu, Haiyang Yu, Fei Huang, and Yongbin Li. 2023 · 2023

Later among the works it cites.

Speculative contrastive decoding

Original

Hongyi Yuan, Keming Lu, Fei Huang, Zheng Yuan, and Chang Zhou. 2023 · 2023

Later among the works it cites.

GLM-130b: An open bilingual pre-trained model

Aohan Zeng, Xiao Liu, Zhengxiao Du, Zihan Wang, Hanyu Lai, Ming Ding, Zhuoyi Yang, Yifan Xu, Wendi Zheng, Xiao Xia, Weng Lam Tam, Zixuan Ma, Yufei Xue, Jidong Zhai, Wenguang Chen, Zhiyuan Liu, Peng Zhang, Yuxiao Dong, and Jie Tang. 2023 · 2023

Later among the works it cites.

Tablegpt: Towards unifying tables, nature language and commands into one gpt

Original

Liangyu Zha, Junlin Zhou, Liyao Li, Rui Wang, Qingyi Huang, Saisai Yang, Jing Yuan, Changbao Su, Xiang Li, Aofeng Su, Tao Zhang, Chen Zhou, Kaizhe Shou, Miao Wang, Wufang Zhu, Guoshan Lu, Chao Ye, Yali Ye, Wentao Ye, Yiming Zhang, Xinglong Deng, Jie Xu, Haobo Wang, Gang Chen, and Junbo Zhao. 2023 · 2023

Later among the works it cites.