AdapterSwap: Continuous Training of LLMs with Data Removal and Access-Control Guarantees

N. Houlsby, A. Giurgiu, S. Jastrzebski, B. Morrone, Q. de Laroussilhe, A. Gesmundo, M. Attariyan, S. Gelly, Parameter-efficient transfer learning for nlp, 2019. arXiv:1902.00751

Original

1902

Earlier work this paper cites.

T. Wolf, L. Debut, V. Sanh, J. Chaumond, C. Delangue, A. Moi, P. Cistac, T. Rault, R. Louf, M. Funtowicz, J. Davison, S. Shleifer, P. von Platen, C. Ma, Y. Jernite, J. Plu, C. Xu, T. L. Scao, S. Gugger, M. Drame, Q. Lhoest, A. M. Rush, Huggingface’s transformers: State-of-the-art natural language processing, 2020. arXiv:1910.03771

Original

1910

Earlier work this paper cites.

Catastrophic interference in connectionist networks: The sequential learning problem,

M. McCloskey, N. J. Cohen, · 1989

Earlier work this paper cites.

P. Lewis, E. Perez, A. Piktus, F. Petroni, V. Karpukhin, N. Goyal, H. Küttler, M. Lewis, W. tau Yih, T. Rocktäschel, S. Riedel, D. Kiela, Retrieval-augmented generation for knowledge-intensive nlp tasks, 2021. arXiv:2005.11401

Original

2005

Earlier work this paper cites.

V. C. Hu, D. Ferraiolo, D. R. Kuhn, et al., Assessment of access control systems, US Department of Commerce, National Institute of Standards and Technology …, 2006

2006

Earlier work this paper cites.

European Parliament, Council of the European Union, Regulation (EU) 2016/679 of the European Parliament and of the Council, 2016. URL: https://data.europa.eu/eli/reg/2016/679/oj

2016

Earlier work this paper cites.

Communication-efficient learning of deep networks from decentralized data,

B. McMahan, E. Moore, D. Ramage, S. Hampson, B. A. y Arcas, · 2017

Earlier work this paper cites.

Simple, scalable adaptation for neural machine translation,

A. Bapna, O. Firat, · 2019

Earlier work this paper cites.

Sentence-BERT: Sentence embeddings using Siamese BERT-networks,

N. Reimers, I. Gurevych, · 2019

Earlier work this paper cites.

Unsupervised domain clusters in pretrained language models,

R. Aharoni, Y. Goldberg, · 2020

Earlier work this paper cites.

Exploring the limits of transfer learning with a unified text-to-text transformer,

C. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y. Zhou, W. Li, P. J. Liu, · 2020

Earlier work this paper cites.

Lora: Low-rank adaptation of large language models,

Original

E. J. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, W. Chen, · 2021

Earlier work this paper cites.

Parameter-efficient transfer learning with diff pruning,

D. Guo, A. Rush, Y. Kim, · 2021

Earlier work this paper cites.

Training neural networks with fixed sparse masks,

Y.-L. Sung, V. Nair, C. A. Raffel, · 2021

Earlier work this paper cites.

The power of scale for parameter-efficient prompt tuning,

B. Lester, R. Al-Rfou, N. Constant, · 2021

Earlier work this paper cites.

Prefix-tuning: Optimizing continuous prompts for generation,

X. L. Li, P. Liang, · 2021

Earlier work this paper cites.

AdapterFusion: Non-destructive task composition for transfer learning,

J. Pfeiffer, A. Kamath, A. Rücklé, K. Cho, I. Gurevych, · 2021

Earlier work this paper cites.

AdapterDrop: On the efficiency of adapters in transformers,

A. Rücklé, G. Geigle, M. Glockner, T. Beck, J. Pfeiffer, N. Reimers, I. Gurevych, · 2021

Cited alongside, same era.

Editing factual knowledge in language models,

N. De Cao, W. Aziz, I. Titov, · 2021

Cited alongside, same era.

The stack: 3 tb of permissively licensed source code,

D. Kocetkov, R. Li, L. Ben Allal, J. Li, C. Mou, C. Muñoz Ferrandis, Y. Jernite, M. Mitchell, S. Hughes, T. Wolf, D. Bahdanau, L. von Werra, H. de Vries, · 2022

Cited alongside, same era.

Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time,

M. Wortsman, G. Ilharco, S. Y. Gadre, R. Roelofs, R. Gontijo-Lopes, A. S. Morcos, H. Namkoong, A. Farhadi, Y. Carmon, S. Kornblith, L. Schmidt, · 2022

Cited alongside, same era.

AdaMix: Mixture-of-adaptations for parameter-efficient model tuning,

Y. Wang, S. Agarwal, S. Mukherjee, X. Liu, J. Gao, A. H. Awadallah, J. Gao, · 2022

Cited alongside, same era.

Cook: Empowering general-purpose language models with modular and collaborative knowledge,

Original

S. Feng, W. Shi, Y. Bai, V. Balachandran, T. He, Y. Tsvetkov, · 2023

Later among the works it cites.

Editing large language models: Problems, methods, and opportunities,

Y. Yao, P. Wang, B. Tian, S. Cheng, Z. Li, S. Deng, H. Chen, N. Zhang, · 2023

Later among the works it cites.

E. Hernandez, B. Z. Li, J. Andreas, Inspecting and editing knowledge representations in language models, 2023. arXiv:2304.00740

Original

2023

Later among the works it cites.

Lost in the middle: How language models use long contexts,

N. F. Liu, K. Lin, J. Hewitt, A. Paranjape, M. Bevilacqua, F. Petroni, P. Liang, · 2023

Later among the works it cites.

Client-customized adaptation for parameter-efficient federated learning,

Y. Kim, J. Kim, W.-L. Mok, J.-H. Park, S. Lee, · 2023

Later among the works it cites.