KronA: Parameter Efficient Tuning with Kronecker Adapter

Exploring the limits of transfer learning with a unified text-to-text transformer

Original

Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2019 · 1910

Earlier work this paper cites.

Huggingface’s transformers: State-of-the-art natural language processing

Original

Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, et al. 2019 · 1910

Earlier work this paper cites.

On the history of the kronecker product

Harold V. Henderson, Friedrich Pukelsheim, and Shayle R. Searle. 1983 · 1983

Earlier work this paper cites.

Adapterfusion: Non-destructive task composition for transfer learning

Original

Jonas Pfeiffer, Aishwarya Kamath, Andreas Rücklé, Kyunghyun Cho, and Iryna Gurevych. 2020 · 2005

Earlier work this paper cites.

Sigmoid-weighted linear units for neural network function approximation in reinforcement learning

Stefan Elfwing, Eiji Uchibe, and Kenji Doya. 2018 · 2018

Earlier work this paper cites.

GLUE: A multi-task benchmark and analysis platform for natural language understanding

Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2018 · 2018

Earlier work this paper cites.

BERT: Pre-training of deep bidirectional transformers for language understanding

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019 · 2019

Earlier work this paper cites.

Parameter-efficient transfer learning for NLP

Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. 2019 · 2019

Earlier work this paper cites.

Language models are unsupervised multitask learners

Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019 · 2019

Cited alongside, same era.

Pushing the limits of rnn compression

Urmish Thakker, Igor Fedorov, Jesse Beu, Dibakar Gope, Chu Zhou, Ganesh Dasika, and Matthew Mattina. 2019 · 2019

Cited alongside, same era.

Exploring versatile generative language model via parameter-efficient transfer learning

Zhaojiang Lin, Andrea Madotto, and Pascale Fung. 2020 · 2020

Cited alongside, same era.

Kronecker decomposition for gpt compression

Original

Ali Edalati, Marzieh Tahaei, Ahmad Rashid, Vahid Partovi Nia, James J Clark, and Mehdi Rezagholizadeh. 2021 · 2021

Cited alongside, same era.

Convolutional neural network compression through generalized kronecker product decomposition

Original

Marawan Gamal Abdel Hameed, Marzieh S Tahaei, Ali Mosleh, and Vahid Partovi Nia. 2021 · 2021

Cited alongside, same era.

Prefix-tuning: Optimizing continuous prompts for generation

Xiang Lisa Li and Percy Liang. 2021 · 2021

Later among the works it cites.

AdapterDrop: On the efficiency of adapters in transformers

Andreas Rücklé, Gregor Geigle, Max Glockner, Tilman Beck, Jonas Pfeiffer, Nils Reimers, and Iryna Gurevych. 2021 · 2021

Later among the works it cites.

Kroneckerbert: Learning kronecker decomposition for pre-trained language models via knowledge distillation

Original

Marzieh S Tahaei, Ella Charlaix, Vahid Partovi Nia, Ali Ghodsi, and Mehdi Rezagholizadeh. 2021 · 2021

Later among the works it cites.

Bitfit: Simple parameter-efficient fine-tuning for transformer-based masked language-models

Original

Elad Ben Zaken, Shauli Ravfogel, and Yoav Goldberg. 2021 · 2021

Later among the works it cites.

Towards a unified view of parameter-efficient transfer learning

Junxian He, Chunting Zhou, Xuezhe Ma, Taylor Berg-Kirkpatrick, and Graham Neubig. 2022 · 2022