Fetching the paper…
Reading the bibliography…
The rising popularity of large foundation models has led to a heightened demand for parameter-efficient fine-tuning methods, such as Low-Rank Adaptation (LoRA), which offer performance comparable to full model fine-tuning while requiring only a few additional parameters tailored to the specific base model.
Distilling the knowledge in a neural network
Geoffrey Hinton · 2015
Earlier work this paper cites.
Sequence-level knowledge distillation
Yoon Kim and Alexander M Rush · 2016
Earlier work this paper cites.
The unreasonable effectiveness of deep features as a perceptual metric
Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang · 2018
Earlier work this paper cites.
Towards understanding knowledge distillation
Phuong Bui Thi Mai and Christoph Lampert · 2019
Earlier work this paper cites.
Relational knowledge distillation
Wonpyo Park, Dongju Kim, Yan Lu, and Minsu Cho · 2019
Earlier work this paper cites.
Be your own teacher: Improve the performance of convolutional neural networks via self distillation
Linfeng Zhang, Jiebo Song, Anni Gao, Jingwei Chen, Chenglong Bao, and Kaisheng Ma · 2019
Earlier work this paper cites.
Denoising diffusion probabilistic models
Jonathan Ho, Ajay Jain, and Pieter Abbeel · 2020
Earlier work this paper cites.
Towards faster and stabilized gan training for high-fidelity few-shot image synthesis
Bingchen Liu, Yizhe Zhu, Kunpeng Song, and Ahmed Elgammal · 2020
Earlier work this paper cites.
Self-distillation as instance-specific label smoothing
Zhilu Zhang and Mert Sabuncu · 2020
Earlier work this paper cites.
Distilling from professors: Enhancing the knowledge distillation of teachers
Duhyeon Bang, Jongwuk Lee, and Hyunjung Shim · 2021
Earlier work this paper cites.
Knowledge distillation: A survey
Jianping Gou, Baosheng Yu, Stephen J Maybank, and Dacheng Tao · 2021
Earlier work this paper cites.
The power of scale for parameter-efficient prompt tuning, 2021
Brian Lester, Rami Al-Rfou, and Noah Constant · 2021
Earlier work this paper cites.
Self-distillation: Towards efficient and compact neural networks
Linfeng Zhang, Chenglong Bao, and Kaisheng Ma · 2021
Earlier work this paper cites.
LoRA: Low-rank adaptation of large language models
Edward J Hu, yelong shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen · 2022
Earlier work this paper cites.
Knowledge distillation: Bad models can be good role models
Gal Kaplun, Eran Malach, Preetum Nakkiran, and Shai Shalev-Shwartz · 2022
Cited alongside, same era.
Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation
Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi · 2022
Cited alongside, same era.
Peft: State-of-the-art parameter-efficient fine-tuning methods
Sourab Mangrulkar, Sylvain Gugger, Lysandre Debut, Younes Belkada, Sayak Paul, and Benjamin Bossan · 2022
Cited alongside, same era.
High-resolution image synthesis with latent diffusion models
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer · 2022
Cited alongside, same era.
Vl-adapter: Parameter-efficient transfer learning for vision-and-language tasks
Yi-Lin Sung, Jaemin Cho, and Mohit Bansal · 2022
Cited alongside, same era.
Efficient knowledge distillation from model checkpoints
Lingling Xu, Haoran Xie, Si-Zhao Joe Qin, Xiaohui Tao, and Fu Lee Wang · 2023
Later among the works it cites.
Llama 3 model card
AI@Meta · 2024
Later among the works it cites.
The claude 3 model family: Opus, sonnet, haiku, 2024
Anthropic · 2024
Later among the works it cites.
Lora-xs: Low-rank adaptation with extremely small number of parameters
Klaudia Bałazy, Mohammadreza Banaei, Karl Aberer, and Jacek Tabor · 2024
Later among the works it cites.
Foura: Fourier low rank adaptation
Shubhankar Borse, Shreya Kadambi, Nilesh Prasad Pandey, Kartikeya Bhardwaj, Viswanath Ganapathy, Sweta Priyadarshi, Risheek Garrepalli, Rafael Esteves, Munawar Hayat, and Fatih Porikli · 2024
Later among the works it cites.
alphaXiv searches the wider corpus for related work and actual follow-ups.
alphaXiv is searching for related work…
Chaofei Wang, Qisen Yang, Rui Huang, Shiji Song, and Gao Huang · 2022
Cited alongside, same era.
Gemini: A family of highly capable multimodal models
Gemini Team, Rohan Anil, and Borgeaud · 2023
Cited alongside, same era.
Svdiff: Compact parameter space for diffusion fine-tuning
Ligong Han, Yinxiao Li, Han Zhang, Peyman Milanfar, Dimitris Metaxas, and Feng Yang · 2023
Cited alongside, same era.
VeRA: Vector-based random matrix adaptation
Dawid J Kopiczko, Tijmen Blankevoort, and Yuki M Asano · 2023
Cited alongside, same era.
Lcm-lora: A universal stable-diffusion acceleration module, 2023
Simian Luo, Yiqin Tan, Suraj Patil, Daniel Gu, Patrick von Platen, Apolinário Passos, Longbo Huang, Jian Li, and Hang Zhao · 2023
Cited alongside, same era.
GPT-4 technical report
OpenAI, Josh Achiam, Steven Adler, and Sandhini Agarwal · 2023
Cited alongside, same era.
X-adapter: Adding universal compatibility of plugins for upgraded diffusion model
Lingmin Ran, Xiaodong Cun, Jia-Wei Liu, Rui Zhao, Song Zijie, Xintao Wang, Jussi Keppo, and Mike Zheng Shou · 2023
Cited alongside, same era.
Implicit style-content separation using b-lora, 2024
Yarden Frenkel, Yael Vinker, Ariel Shamir, and Daniel Cohen-Or · 2024
Later among the works it cites.
Progressive knowledge distillation of stable diffusion xl using layer level loss, 2024
Yatharth Gupta, Vishnu V. Jaddipal, Harish Prabhala, Sayak Paul, and Patrick Von Platen · 2024
Later among the works it cites.
Svft: Parameter-efficient fine-tuning with singular vectors
Vijay Lingam, Atula Tejaswi, Aditya Vavre, Aneesh Shetty, Gautham Krishna Gudur, Joydeep Ghosh, Alex Dimakis, Eunsol Choi, Aleksandar Bojchevski, and Sujay Sanghavi · 2024
Later among the works it cites.
Dora: Weight-decomposed low-rank adaptation
Shih-Yang Liu, Chien-Yi Wang, Hongxu Yin, Pavlo Molchanov, Yu-Chiang Frank Wang, Kwang-Ting Cheng, and Min-Hung Chen · 2024
Later among the works it cites.
Pissa: Principal singular values and singular vectors adaptation of large language models
Fanxu Meng, Zhaohui Wang, and Muhan Zhang · 2024
Later among the works it cites.
DINOv2: Learning robust visual features without supervision
Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy V. Vo, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel HAZIZA, Francisco Massa, Alaaeldin El-Nouby, Mido Assran, Nicolas Ballas, Wojciech Galuba, Russell Howes, Po-Yao Huang, Shang-Wen Li, Ishan Misra, Michael Rabbat, Vasu Sharma, Gabriel Synnaeve, Hu Xu, Herve Jegou, Julien Mairal, Patrick Labatut, Armand Joulin, and Piotr Bojanowski · 2024
Later among the works it cites.
SDXL: Improving latent diffusion models for high-resolution image synthesis
Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach · 2024
Later among the works it cites.
Trans-LoRA
Runqian Wang, Soumya Ghosh, David Cox, Diego Antognini, Aude Oliva, Rogerio Feris, and Leonid Karlinsky · 2024
Later among the works it cites.
Tinyllama: An open-source small language model, 2024
Peiyuan Zhang, Guangtao Zeng, Tianduo Wang, and Wei Lu · 2024
Later among the works it cites.