Mitigating Catastrophic Forgetting in Language Transfer via Model Merging

Huggingface’s transformers: State-of-the-art natural language processing

Original

Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, and Jamie Brew. 2019 · 1910

Earlier work this paper cites.

Animating rotation with quaternion curves

Ken Shoemake. 1985 · 1985

Earlier work this paper cites.

Catastrophic forgetting in connectionist networks

Robert M French. 1999 · 1999

Earlier work this paper cites.

Europarl: A parallel corpus for statistical machine translation

Philipp Koehn. 2005 · 2005

Earlier work this paper cites.

Query-driven active surveying for collective classification

Galileo Mark Namata, Ben London, Lise Getoor, and Bert Huang. 2012 · 2012

Earlier work this paper cites.

An empirical investigation of catastrophic forgeting in gradient-based neural networks

Original

Ian J. Goodfellow, Mehdi Mirza, Xia Da, Aaron C. Courville, and Yoshua Bengio. 2014 · 2014

Earlier work this paper cites.

European Union language resources in Sketch Engine

Vít Baisa, Jan Michelfeit, Marek Medveď, and Miloš Jakubíček. 2016 · 2016

Earlier work this paper cites.

TriviaQA: A large scale distantly supervised challenge dataset for reading comprehension

Mandar Joshi, Eunsol Choi, Daniel Weld, and Luke Zettlemoyer. 2017 · 2017

Earlier work this paper cites.

Communication-efficient learning of deep networks from decentralized data

Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Agüera y Arcas. 2017 · 2017

Earlier work this paper cites.

Think you have solved question answering? try arc, the ai2 reasoning challenge

Original

Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. 2018 · 2018

Earlier work this paper cites.

XNLI: Evaluating cross-lingual sentence representations

Alexis Conneau, Ruty Rinott, Guillaume Lample, Adina Williams, Samuel Bowman, Holger Schwenk, and Veselin Stoyanov. 2018 · 2018

Earlier work this paper cites.

Measuring catastrophic forgetting in neural networks

Ronald Kemker, Marc McClure, Angelina Abitino, Tyler L. Hayes, and Christopher Kanan. 2018 · 2018

Earlier work this paper cites.

SentencePiece: A simple and language independent subword tokenizer and detokenizer for neural text processing

Taku Kudo and John Richardson. 2018 · 2018

Earlier work this paper cites.

MathQA: Towards interpretable math word problem solving with operation-based formalisms

Aida Amini, Saadia Gabriel, Shanchuan Lin, Rik Koncel-Kedziorski, Yejin Choi, and Hannaneh Hajishirzi. 2019 · 2019

Earlier work this paper cites.

Openwebtext corpus

Aaron Gokaslan, Vanya Cohen, Ellie Pavlick, and Stefanie Tellex. 2019 · 2019

Earlier work this paper cites.

Pytorch: An imperative style, high-performance deep learning library

Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Köpf, Edward Z. Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. 2019 · 2019

Earlier work this paper cites.

Language models are unsupervised multitask learners

Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019 · 2019

Earlier work this paper cites.

Experience replay for continual learning

David Rolnick, Arun Ahuja, Jonathan Schwarz, Timothy P. Lillicrap, and Gregory Wayne. 2019 · 2019

Earlier work this paper cites.

PAWS-X: A cross-lingual adversarial dataset for paraphrase identification

Yinfei Yang, Yuan Zhang, Chris Tar, and Jason Baldridge. 2019 · 2019

Earlier work this paper cites.

HellaSwag: Can a machine really finish your sentence?

Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. 2019 · 2019

Earlier work this paper cites.

EXAMS: A multi-subject high school examinations dataset for cross-lingual and multilingual question answering

Momchil Hardalov, Todor Mihaylov, Dimitrina Zlatkova, Yoan Dinkov, Ivan Koychev, and Preslav Nakov. 2020 · 2020

Earlier work this paper cites.

Mixout: Effective regularization to finetune large-scale pretrained language models

Cheolhyoung Lee, Kyunghyun Cho, and Wanmo Kang. 2020 · 2020

Earlier work this paper cites.

Deepspeed: System optimizations enable training deep learning models with over 100 billion parameters

Jeff Rasley, Samyam Rajbhandari, Olatunji Ruwase, and Yuxiong He. 2020 · 2020

Earlier work this paper cites.

Training verifiers to solve math word problems

Original

Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. 2021 · 2021

Earlier work this paper cites.

Measuring massive multitask language understanding

Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2021 · 2021

Earlier work this paper cites.

Winogrande: an adversarial winograd schema challenge at scale

Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. 2021 · 2021

Earlier work this paper cites.

mT5: A massively multilingual pre-trained text-to-text transformer

Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. 2021 · 2021

Earlier work this paper cites.

Fusing finetuned models for better pretraining

Original

Leshem Choshen, Elad Venezian, Noam Slonim, and Yoav Katz. 2022 · 2022

Earlier work this paper cites.

How does gpt obtain its ability? tracing emergent abilities of language models to their sources

Hao Fu, Yao; Peng and Tushar Khot. 2022 · 2022

Earlier work this paper cites.

Fast vocabulary transfer for language model compression

Leonidas Gee, Andrea Zugarini, Leonardo Rigutini, and Paolo Torroni. 2022 · 2022

Earlier work this paper cites.

LoRA: Low-rank adaptation of large language models

Edward J Hu, yelong shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2022 · 2022

Cited alongside, same era.

Branch-train-merge: Embarrassingly parallel training of expert language models

Margaret Li, Suchin Gururangan, Tim Dettmers, Mike Lewis, Tim Althoff, Noah A. Smith, and Luke Zettlemoyer. 2022 · 2022

Cited alongside, same era.

Holistic evaluation of language models

Original

Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga, Yian Zhang, Deepak Narayanan, Yuhuai Wu, Ananya Kumar, Benjamin Newman, Binhang Yuan, Bobby Yan, Ce Zhang, Christian Cosgrove, Christopher D. Manning, Christopher Ré, Diana Acosta-Navas, Drew A. Hudson, Eric Zelikman, Esin Durmus, Faisal Ladhak, Frieda Rong, Hongyu Ren, Huaxiu Yao, Jue Wang, Keshav Santhanam, Laurel J. Orr, Lucia Zheng, Mert Yüksekgönül, Mirac Suzgun, Nathan Kim, Neel Guha, Niladri S. Chatterji, Omar Khattab, Peter Henderson, Qian Huang, Ryan Chi, Sang Michael Xie, Shibani Santurkar, Surya Ganguli, Tatsunori Hashimoto, Thomas Icard, Tianyi Zhang, Vishrav Chaudhary, William Wang, Xuechen Li, Yifan Mai, Yuhui Zhang, and Yuta Koreeda. 2022 · 2022

Cited alongside, same era.

Merging models with fisher-weighted averaging

Michael Matena and Colin Raffel. 2022 · 2022

Cited alongside, same era.

Openhermes 2.5: An open dataset of synthetic data for generalist llm assistants

Teknium. 2023 · 2023

Later among the works it cites.

D4: improving LLM pretraining via document de-duplication and diversification

Kushal Tirumala, Daniel Simig, Armen Aghajanyan, and Ari Morcos. 2023 · 2023

Later among the works it cites.

Redpajama: an open dataset for training large language models

Together.ai. 2023 · 2023

Later among the works it cites.

Llama: Open and efficient foundation language models

Original

Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurélien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023 · 2023

Later among the works it cites.

Overcoming catastrophic forgetting in massively multilingual continual learning

Genta Indra Winata, Lingjue Xie, Karthik Radhakrishnan, Shijie Wu, Xisen Jin, Pengxiang Cheng, Mayank Kulkarni, and Daniel Preotiuc-Pietro. 2023 · 2023

Later among the works it cites.