2023

In-context Example Selection with Influences

Nguyen, Tai, Wong, Eric

Understand

In-context learning (ICL) is a powerful paradigm emerged from large language models (LLMs).

Despite its promises, ICL performance is known to be highly sensitive to input examples.
In this work, we use $\textit{in-context influences}$ to analyze few-shot ICL performance directly from the in-context examples.
Our proposed influence-based example selection method can identify both positive and negative examples, outperforming several baselines when evaluated on 9 SuperGLUE tasks.

Built on

“Understanding black-box predictions via influence functions”
Pang Koh and Percy Liang · 2017
Earlier work this paper cites.
“Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge”
Peter Clark et al · 2018
Earlier work this paper cites.
“Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answering”
Todor Mihaylov, Peter Clark, Tushar Khot and Ashish Sabharwal · 2018
Earlier work this paper cites.
“BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding”
Jacob Devlin, Ming-Wei Chang, Kenton Lee and Kristina Toutanova · 2019
Earlier work this paper cites.
“Data Shapley: Equitable Valuation of Data for Machine Learning”
Amirata Ghorbani and James. Zou · 2019
Earlier work this paper cites.
“On the Accuracy of Influence Functions for Measuring Group Effects”
Pang Koh, Kai-Siang Ang, Hubert.. Teo and Percy Liang · 2019
Earlier work this paper cites.
“RoBERTa: A Robustly Optimized BERT Pretraining Approach”
Yinhan Liu et al · 2019
Earlier work this paper cites.
“Language models are unsupervised multitask learners”
Alec Radford et al · 2019
Earlier work this paper cites.
“Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks”
Nils Reimers and Iryna Gurevych · 2019
Earlier work this paper cites.
“SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems”
Alex Wang et al · 2019
Earlier work this paper cites.
“HellaSwag: Can a Machine Really Finish Your Sentence?”
Rowan Zellers et al · 2019
Earlier work this paper cites.
“Language Models are Few-Shot Learners”
Tom. Brown et al · 2020
Earlier work this paper cites.
“Explaining Black Box Predictions and Unveiling Data Artifacts through Influence Functions”
Xiaochuang Han, Byron. Wallace and Yulia Tsvetkov · 2020
Earlier work this paper cites.

Similar

“Estimating Training Data Influence by Tracing Gradient Descent”
Garima Pruthi, Frederick Liu, Satyen Kale and Mukund Sundararajan · 2020
Cited alongside, same era.
“Surface Form Competition: Why the Highest Probability Answer Isn’t Always Right”
Ari Holtzman et al · 2021
Cited alongside, same era.
“Reordering Examples Helps during Priming-based Few-Shot Learning”
Sawan Kumar and Partha Talukdar · 2021
Cited alongside, same era.
“True Few-Shot Learning with Language Models”
Original
Ethan Perez, Douwe Kiela and Kyunghyun Cho · 2021
Cited alongside, same era.
“Calibrate Before Use: Improving Few-shot Performance of Language Models”
Zihao Zhao et al · 2021
Cited alongside, same era.
“What learning algorithm is in-context learning? Investigations with linear models”
Ekin Akyürek et al · 2022
Cited alongside, same era.
“GPT-NeoX-20B: An Open-Source Autoregressive Language Model”
Sidney Black et al · 2022
Cited alongside, same era.
“On the Relation between Sensitivity and Accuracy in In-context Learning”, 2022
Yanda Chen et al · 2022
Cited alongside, same era.
“Careful Data Curation Stabilizes In-context Learning”
Ting-Yun Chang and Robin Jia · 2022
Cited alongside, same era.
“Demystifying Prompts in Language Models via Perplexity Estimation”, 2022
Hila Gonen et al · 2022
Cited alongside, same era.
“Datamodels: Predicting Predictions from Training Data”, 2022
Andrew Ilyas et al · 2022
Cited alongside, same era.
“What Makes Good In-Context Examples for GPT-3?”
Jiachang Liu et al · 2022
Cited alongside, same era.

Then

“Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivity”
Yao Lu et al · 2022
Later among the works it cites.
“Noisy Channel Language Model Prompting for Few-Shot Text Classification”
Sewon Min, Mike Lewis, Hannaneh Hajishirzi and Luke Zettlemoyer · 2022
Later among the works it cites.
“Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?”, 2022
Sewon Min et al · 2022
Later among the works it cites.
“Show Your Work: Scratchpads for Intermediate Computation with Language Models”, 2022
Maxwell Nye et al · 2022
Later among the works it cites.
“Impact of Pretraining Term Frequencies on Few-Shot Reasoning”
Yasaman Razeghi, Robert. Logan, Matt Gardner and Sameer Singh · 2022
Later among the works it cites.
“Learning To Retrieve Prompts for In-Context Learning”
Ohad Rubin, Jonathan Herzig and Jonathan Berant · 2022
Later among the works it cites.
“Selective Annotation Makes Language Models Better Few-Shot Learners”
Hongjin Su et al · 2022
Later among the works it cites.
“Chain of Thought Prompting Elicits Reasoning in Large Language Models”
Jason Wei et al · 2022
Later among the works it cites.
“OPT: Open Pre-trained Transformer Language Models”, 2022
Susan Zhang et al · 2022
Later among the works it cites.
“TEMPERA: Test-Time Prompting via Reinforcement Learning”
Original
Tianjun Zhang et al · 2022
Later among the works it cites.
“Faithful Chain-of-Thought Reasoning”
Qing Lyu et al · 2023
Closest in time.
“LLaMA: Open and Efficient Foundation Language Models”
Original
Hugo Touvron et al · 2023
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…