Outlier-weighed Layerwise Sampling for LLM Fine-tuning

Language models are few-shot learners

Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020 · 1901

Earlier work this paper cites.

Bayesian estimates of equation system parameters: an application of integration by monte carlo

Teun Kloek and Herman K Van Dijk. 1978 · 1978

Earlier work this paper cites.

Measuring massive multitask language understanding

Original

Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2020 · 2009

Earlier work this paper cites.

Stochastic optimization with importance sampling for regularized loss minimization

Peilin Zhao and Tong Zhang. 2015 · 2015

Earlier work this paper cites.

Freezeout: Accelerate training by progressively freezing layers

Original

Andrew Brock, Theodore Lim, James M Ritchie, and Nick Weston. 2017 · 2017

Earlier work this paper cites.

Parameter-efficient transfer learning for nlp

Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. 2019 · 2019

Earlier work this paper cites.

Warp: Word-level adversarial reprogramming

Original

Karen Hambardzumyan, Hrant Khachatrian, and Jonathan May. 2021 · 2021

Earlier work this paper cites.

Towards a unified view of parameter-efficient transfer learning

Original

Junxian He, Chunting Zhou, Xuezhe Ma, Taylor Berg-Kirkpatrick, and Graham Neubig. 2021 · 2021

Earlier work this paper cites.

Lora: Low-rank adaptation of large language models

Original

Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2021 · 2021

Earlier work this paper cites.

Bert busters: Outlier dimensions that disrupt transformers

Original

Olga Kovaleva, Saurabh Kulshreshtha, Anna Rogers, and Anna Rumshisky. 2021 · 2021

Earlier work this paper cites.

The power of scale for parameter-efficient prompt tuning

Original

Brian Lester, Rami Al-Rfou, and Noah Constant. 2021 · 2021

Earlier work this paper cites.

Prefix-tuning: Optimizing continuous prompts for generation

Original

Xiang Lisa Li and Percy Liang. 2021 · 2021

Earlier work this paper cites.

Parameter-efficient multi-task fine-tuning for transformers via shared hypernetworks

Original

Rabeeh Karimi Mahabadi, Sebastian Ruder, Mostafa Dehghani, and James Henderson. 2021 · 2021

Earlier work this paper cites.

Factual probing is [mask]: Learning vs. learning to recall

Original

Zexuan Zhong, Dan Friedman, and Danqi Chen. 2021 · 2021

Earlier work this paper cites.

Llm. int8 (): 8-bit matrix multiplication for transformers at scale

Tim Dettmers, Mike Lewis, Younes Belkada, and Luke Zettlemoyer. 2022 · 2022

Earlier work this paper cites.

Black-box prompt learning for pre-trained language models

Original

Shizhe Diao, Zhichao Huang, Ruijia Xu, Xuechun Li, Yong Lin, Xiao Zhou, and Tong Zhang. 2022 · 2022

Cited alongside, same era.

Outliers dimensions that disrupt transformers are driven by frequency

Original

Giovanni Puccetti, Anna Rogers, Aleksandr Drozd, and Felice Dell’Orletta. 2022 · 2022

Cited alongside, same era.

Palm 2 technical report

Original

Rohan Anil, Andrew M Dai, Orhan Firat, Melvin Johnson, Dmitry Lepikhin, Alexandre Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng Chen, et al. 2023 · 2023

Cited alongside, same era.

Llm-adapters: An adapter family for parameter-efficient fine-tuning of large language models

Original

Zhiqiang Hu, Lei Wang, Yihuai Lan, Wanyu Xu, Ee-Peng Lim, Lidong Bing, Xing Xu, Soujanya Poria, and Roy Ka-Wei Lee. 2023 · 2023

Cited alongside, same era.

Is chatgpt the ultimate programming assistant–how far is it?

Original

Haoye Tian, Weiqi Lu, Tsz On Li, Xunzhu Tang, Shing-Chi Cheung, Jacques Klein, and Tegawendé F Bissyandé. 2023 · 2023

Later among the works it cites.

Llama 2: Open foundation and fine-tuned chat models

Original

Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023 · 2023

Later among the works it cites.

Adaptive budget allocation for parameter-efficient fine-tuning

Qingru Zhang, Minshuo Chen, Alexander Bukharin, Pengcheng He, Yu Cheng, Weizhu Chen, and Tuo Zhao. 2023 · 2023

Later among the works it cites.

Lora learns less and forgets less

Original

Dan Biderman, Jose Gonzalez Ortiz, Jacob Portes, Mansheej Paul, Philip Greengard, Connor Jennings, Daniel King, Sam Havens, Vitaliy Chiley, Jonathan Frankle, et al. 2024 · 2024

Closest in time.