Do Influence Functions Work on Large Language Models?

Language models are few-shot learners

Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020 · 1901

Earlier work this paper cites.

The influence curve and its role in robust estimation

Frank R Hampel. 1974 · 1974

Earlier work this paper cites.

Residuals and influence in regression

Robert F Ling. 1984 · 1984

Earlier work this paper cites.

Scaling laws for neural language models

Original

Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. 2020 · 2001

Earlier work this paper cites.

Prediction error estimation: a comparison of resampling methods

Annette M Molinaro, Richard Simon, and Ruth M Pfeiffer. 2005 · 2005

Earlier work this paper cites.

Deep learning via hessian-free optimization

James Martens et al. 2010 · 2010

Earlier work this paper cites.

Fastif: Scalable influence functions for efficient model interpretation and debugging

Original

Han Guo, Nazneen Fatema Rajani, Peter Hase, Mohit Bansal, and Caiming Xiong. 2020 · 2012

Earlier work this paper cites.

Approximate matrix inversion for high-throughput data detection in the large-scale mimo uplink

Michael Wu, Bei Yin, Aida Vosoughi, Christoph Studer, Joseph R Cavallaro, and Chris Dick. 2013 · 2013

Earlier work this paper cites.

Second-order stochastic optimization for machine learning in linear time

Naman Agarwal, Brian Bullins, and Elad Hazan. 2017 · 2017

Earlier work this paper cites.

Understanding black-box predictions via influence functions

Pang Wei Koh and Percy Liang. 2017 · 2017

Earlier work this paper cites.

Universal and transferable adversarial attacks on aligned language models

Original

Andy Zou, Zifan Wang, J Zico Kolter, and Matt Fredrikson. 2023b · 2017

Earlier work this paper cites.

Visualizing the loss landscape of neural nets

Hao Li, Zheng Xu, Gavin Taylor, Christoph Studer, and Tom Goldstein. 2018 · 2018

Earlier work this paper cites.

Carer: Contextualized affect representations for emotion recognition

Elvis Saravia, Hsien-Chi Toby Liu, Yen-Hao Huang, Junlin Wu, and Yi-Shin Chen. 2018 · 2018

Earlier work this paper cites.

Input similarity from the neural network perspective

Guillaume Charpiat, Nicolas Girard, Loris Felardos, and Yuliya Tarabalka. 2019 · 2019

Earlier work this paper cites.

Data shapley: Equitable valuation of data for machine learning

Amirata Ghorbani and James Zou. 2019 · 2019

Earlier work this paper cites.

Estimating training data influence by tracing gradient descent

Garima Pruthi, Frederick Liu, Satyen Kale, and Mukund Sundararajan. 2020 · 2020

Earlier work this paper cites.

Scaling laws for transfer

Original

Danny Hernandez, Jared Kaplan, Tom Henighan, and Sam McCandlish. 2021 · 2021

Earlier work this paper cites.

Lora: Low-rank adaptation of large language models

Original

Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2021 · 2021

Cited alongside, same era.

Beta shapley: a unified and noise-reduced data valuation framework for machine learning

Original

Yongchan Kwon and James Zou. 2021 · 2021

Cited alongside, same era.

If influence functions are the answer, then what is the question?

Juhan Bae, Nathan Ng, Alston Lo, Marzyeh Ghassemi, and Roger B Grosse. 2022 · 2022

Cited alongside, same era.

Training a helpful and harmless assistant with reinforcement learning from human feedback

Original

Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. 2022 · 2022

Cited alongside, same era.

Peft: State-of-the-art parameter-efficient fine-tuning methods

A simple and effective pruning approach for large language models

Original

Mingjie Sun, Zhuang Liu, Anna Bair, and J Zico Kolter. 2023 · 2023

Later among the works it cites.

Stanford alpaca: An instruction-following llama model

Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B Hashimoto. 2023 · 2023

Later among the works it cites.

Llama 2: Open foundation and fine-tuned chat models

Original

Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023 · 2023

Later among the works it cites.

Aligning large language models with human: A survey

Original

Yufei Wang, Wanjun Zhong, Liangyou Li, Fei Mi, Xingshan Zeng, Wenyong Huang, Lifeng Shang, Xin Jiang, and Qun Liu. 2023 · 2023

Later among the works it cites.

What is your data worth to gpt? llm-scale data valuation with influence functions