MEFT: Memory-Efficient Fine-Tuning through Sparse Adapter

Roberta: A robustly optimized bert pretraining approach

Original

Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019 · 1907

Earlier work this paper cites.

End-to-end memory networks

Sainbayar Sukhbaatar, Jason Weston, Rob Fergus, et al. 2015 · 2015

Earlier work this paper cites.

SQuAD: 100,000+ questions for machine comprehension of text

Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016 · 2016

Earlier work this paper cites.

Outrageously large neural networks: The sparsely-gated mixture-of-experts layer

Noam Shazeer, *Azalia Mirhoseini, *Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean. 2017 · 2017

Earlier work this paper cites.

Efficient and robust approximate nearest neighbor search using hierarchical navigable small world graphs

Yu A Malkov and Dmitry A Yashunin. 2018 · 2018

Earlier work this paper cites.

Parameter-efficient transfer learning for nlp

Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. 2019 · 2019

Earlier work this paper cites.

Billion-scale similarity search with GPUs

Jeff Johnson, Matthijs Douze, and Hervé Jégou. 2019 · 2019

Earlier work this paper cites.

Natural questions: a benchmark for question answering research

Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, et al. 2019 · 2019

Earlier work this paper cites.

Language models are few-shot learners

Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020 · 2020

Earlier work this paper cites.

Slide : In defense of smart algorithms over hardware acceleration for large-scale deep learning systems

Original

Beidi Chen, Tharun Medini, James Farwell, Sameh Gobriel, Charlie Tai, and Anshumali Shrivastava. 2020 · 2020

Earlier work this paper cites.

Accelerating large-scale inference with anisotropic vector quantization

Ruiqi Guo, Philip Sun, Erik Lindgren, Quan Geng, David Simcha, Felix Chern, and Sanjiv Kumar. 2020 · 2020

Earlier work this paper cites.

Inducing and exploiting activation sparsity for fast inference on deep neural networks

Mark Kurtz, Justin Kopinsky, Rati Gelashvili, Alexander Matveev, John Carr, Michael Goin, William Leiserson, Sage Moore, Nir Shavit, and Dan Alistarh. 2020 · 2020

Earlier work this paper cites.

{MONGOOSE}: A learnable {lsh} framework for efficient neural network training

Beidi Chen, Zichang Liu, Binghui Peng, Zhaozhuo Xu, Jonathan Lingjie Li, Tri Dao, Zhao Song, Anshumali Shrivastava, and Christopher Re. 2021 · 2021

Earlier work this paper cites.

Training verifiers to solve math word problems

Original

Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. 2021 · 2021

Earlier work this paper cites.

Making pre-trained language models better few-shot learners

Tianyu Gao, Adam Fisch, and Danqi Chen. 2021 · 2021

Earlier work this paper cites.

Transformer feed-forward layers are key-value memories

Mor Geva, Roei Schuster, Jonathan Berant, and Omer Levy. 2021 · 2021

Earlier work this paper cites.

Prefix-tuning: Optimizing continuous prompts for generation

Xiang Lisa Li and Percy Liang. 2021 · 2021

Cited alongside, same era.

AdapterFusion: Non-destructive task composition for transfer learning

Jonas Pfeiffer, Aishwarya Kamath, Andreas Rücklé, Kyunghyun Cho, and Iryna Gurevych. 2021 · 2021

Cited alongside, same era.

{ \{ ZeRO-Offload } \} : Democratizing { \{ Billion-Scale } \} model training

Jie Ren, Samyam Rajbhandari, Reza Yazdani Aminabadi, Olatunji Ruwase, Shuangyan Yang, Minjia Zhang, Dong Li, and Yuxiong He. 2021 · 2021

Cited alongside, same era.

It’s not just size that matters: Small language models are also few-shot learners

Timo Schick and Hinrich Schütze. 2021 · 2021

Cited alongside, same era.

Training neural networks with fixed sparse masks

Yi-Lin Sung, Varun Nair, and Colin A Raffel. 2021 · 2021

Cited alongside, same era.

Can generative pre-trained language models serve as knowledge bases for closed-book QA?

AdaMix: Mixture-of-adaptations for parameter-efficient model tuning

Yaqing Wang, Sahaj Agarwal, Subhabrata Mukherjee, Xiaodong Liu, Jing Gao, Ahmed Hassan Awadallah, and Jianfeng Gao. 2022 · 2022

Later among the works it cites.

Model leeching: An extraction attack targeting llms

Lewis Birch, William Hackett, Stefan Trawicki, Neeraj Suri, and Peter Garraghan. 2023 · 2023

Later among the works it cites.

Qlora: Efficient finetuning of quantized llms

Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. 2023 · 2023

Later among the works it cites.

Mixture-of-domain-adapters: Decoupling and injecting domain knowledge to pre-trained language models’ memories

Shizhe Diao, Tianyang Xu, Ruijia Xu, Jiawei Wang, and Tong Zhang. 2023 · 2023

Later among the works it cites.

Sparsegpt: Massive language models can be accurately pruned in one-shot

Elias Frantar and Dan Alistarh. 2023 · 2023

Later among the works it cites.