Fetching the paper…
Reading the bibliography…
Large Language Models (LLMs) have demonstrated remarkable in-context learning (ICL) capabilities.
Superposition of many models into one, June 2019
Brian Cheung, Alex Terekhov, Yubei Chen, Pulkit Agrawal, and Bruno Olshausen · 1902
Earlier work this paper cites.
Language models are unsupervised multitask learners
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al · 2019
Earlier work this paper cites.
Language Models are Few-Shot Learners
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei · 2020
Earlier work this paper cites.
Language models are multiverse generators, January 2021
moire · 2021
Earlier work this paper cites.
Multiversal views on language models, February 2021
Laria Reynolds and Kyle McDonell · 2021
Earlier work this paper cites.
Data distributional properties drive emergent in-context learning in transformers
Stephanie Chan, Adam Santoro, Andrew Lampinen, Jane Wang, Aaditya Singh, Pierre Richemond, James McClelland, and Felix Hill · 2022
Earlier work this paper cites.
Why can gpt learn in-context? language models secretly perform gradient descent as meta optimizers
Damai Dai, Yutao Sun, Li Dong, Yaru Hao, Zhifang Sui, and Furu Wei · 2022
Earlier work this paper cites.
Lift: Language-interfaced fine-tuning for non-language machine learning tasks
Tuan Dinh, Yuchen Zeng, Ruisu Zhang, Ziqian Lin, Shashank Rajput, Michael Gira, Jy-yong Sohn, Dimitris Papailiopoulos, and Kangwook Lee · 2022
Earlier work this paper cites.
Toy Models of Superposition, September 2022
Nelson Elhage, Tristan Hume, Catherine Olsson, Nicholas Schiefer, Tom Henighan, Shauna Kravec, Zac Hatfield-Dodds, Robert Lasenby, Dawn Drain, Carol Chen, Roger Grosse, Sam McCandlish, Jared Kaplan, Dario Amodei, Martin Wattenberg, and Christopher Olah · 2022
Earlier work this paper cites.
Simulators, 2022
Janus · 2022
Earlier work this paper cites.
Can language models learn from explanations in context?
Andrew K Lampinen, Ishita Dasgupta, Stephanie CY Chan, Kory Matthewson, Michael Henry Tessler, Antonia Creswell, James L McClelland, Jane X Wang, and Felix Hill · 2022
Earlier work this paper cites.
In-context reinforcement learning with algorithm distillation
Michael Laskin, Luyu Wang, Junhyuk Oh, Emilio Parisotto, Stephen Spencer, Richie Steigerwald, DJ Strouse, Steven Hansen, Angelos Filos, Ethan Brooks, et al · 2022
Earlier work this paper cites.
Z-icl: Zero-shot in-context learning with pseudo-demonstrations
Xinxi Lyu, Sewon Min, Iz Beltagy, Luke Zettlemoyer, and Hannaneh Hajishirzi · 2022
Earlier work this paper cites.
Rethinking the role of demonstrations: What makes in-context learning work?
Sewon Min, Xinxi Lyu, Ari Holtzman, Mikel Artetxe, Mike Lewis, Hannaneh Hajishirzi, and Luke Zettlemoyer · 2022
Earlier work this paper cites.
Datamux: Data multiplexing for neural networks
Vishvak Murahari, Carlos E. Jimenez, Runzhe Yang, and Karthik Narasimhan · 2022
Earlier work this paper cites.
In-context learning and induction heads
Catherine Olsson, Nelson Elhage, Neel Nanda, Nicholas Joseph, Nova DasSarma, Tom Henighan, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, et al · 2022
Earlier work this paper cites.
Statistically meaningful approximation: a case study on approximating turing machines with transformers
Colin Wei, Yining Chen, and Tengyu Ma · 2022
Earlier work this paper cites.
An explanation of in-context learning as implicit bayesian inference
Sang Michael Xie, Aditi Raghunathan, Percy Liang, and Tengyu Ma · 2022
Cited alongside, same era.
Teaching algorithmic reasoning via in-context learning
Hattie Zhou, Azade Nova, Hugo Larochelle, Aaron Courville, Behnam Neyshabur, and Hanie Sedghi · 2022
Cited alongside, same era.
Transformers learn to implement preconditioned gradient descent for in-context learning
Kwangjun Ahn, Xiang Cheng, Hadi Daneshmand, and Suvrit Sra · 2023
Cited alongside, same era.
What learning algorithm is in-context learning? Investigations with linear models, May 2023
Ekin Akyürek, Dale Schuurmans, Jacob Andreas, Tengyu Ma, and Denny Zhou · 2023
Cited alongside, same era.
Looped transformers as programmable computers, 2023
Yufeng Zhang, Fengzhuo Zhang, Zhuoran Yang, and Zhaoran Wang · 2023
Later among the works it cites.
What algorithms can transformers learn? a study in length generalization
Hattie Zhou, Arwen Bradley, Etai Littwin, Noam Razin, Omid Saremi, Josh Susskind, Samy Bengio, and Preetum Nakkiran · 2023
Later among the works it cites.
Many-shot in-context learning, 2024
Rishabh Agarwal, Avi Singh, Lei M. Zhang, Bernd Bohnet, Stephanie Chan, Ankesh Anand, Zaheer Abbas, Azade Nova, John D. Co-Reyes, Eric Chu, Feryal Behbahani, Aleksandra Faust, and Hugo Larochelle · 2024
Closest in time.
Llama 3 model card
AI@Meta · 2024
Closest in time.
Identifying and analyzing task-encoding tokens in large language models
Yu Bai, Heyan Huang, Cesare Spinoso-Di Piano, Marc-Antoine Rondeau, Sanxing Chen, Yang Gao, and Jackie Chi Kit Cheung · 2024
alphaXiv searches the wider corpus for related work and actual follow-ups.
alphaXiv is searching for related work…
Angeliki Giannou, Shashank Rajput, Jy-yong Sohn, Kangwook Lee, Jason D. Lee, and Dimitris Papailiopoulos · 2023
Cited alongside, same era.
Mamba: Linear-time sequence modeling with selective state spaces
Albert Gu and Tri Dao · 2023
Cited alongside, same era.
In-Context Learning Creates Task Vectors, October 2023
Roee Hendel, Mor Geva, and Amir Globerson · 2023
Cited alongside, same era.
Editing Models with Task Arithmetic, March 2023
Gabriel Ilharco, Marco Tulio Ribeiro, Mitchell Wortsman, Suchin Gururangan, Ludwig Schmidt, Hannaneh Hajishirzi, and Ali Farhadi · 2023
Cited alongside, same era.
The waluigi effect (mega-post), 2023
Cleo Nardo · 2023
Cited alongside, same era.
Transformers Learn In-Context by Gradient Descent
Johannes Von Oswald, Eyvind Niklasson, Ettore Randazzo, Joao Sacramento, Alexander Mordvintsev, Andrey Zhmoginov, and Max Vladymyrov · 2023
Cited alongside, same era.
In-context learning through the bayesian prism, June 2023
Madhur Panwar, Kabir Ahuja, and Navin Goyal · 2023
Cited alongside, same era.
The effects of pretraining task diversity on in-context learning of ridge regression
Allan Raventos, Mansheej Paul, Feng Chen, and Surya Ganguli · 2023
Cited alongside, same era.
Closest in time.
Picle: Eliciting diverse behaviors from large language models with persona in-context learning
Hyeong Kyu Choi and Yixuan Li · 2024
Closest in time.
Is in-context learning in large language models bayesian? a martingale perspective
Fabian Falck, Ziyu Wang, and Chris Holmes · 2024
Closest in time.
Is mamba capable of in-context learning?
Riccardo Grazzi, Julien Siems, Simon Schrodi, Thomas Brox, and Frank Hutter · 2024
Closest in time.
Long-context llms struggle with long in-context learning
Tianle Li, Ge Zhang, Quy Duc Do, Xiang Yue, and Wenhu Chen · 2024
Closest in time.
Dual Operating Modes of In-Context Learning, February 2024
Ziqian Lin and Kangwook Lee · 2024
Closest in time.
Sheng Liu, Haotian Ye, Lei Xing, and James Zou · 2024
Closest in time.
Can mamba learn how to learn? a comparative study on in-context learning tasks
Jongho Park, Jaeseung Park, Zheyang Xiong, Nayoung Lee, Jaewoong Cho, Samet Oymak, Kangwook Lee, and Dimitris Papailiopoulos · 2024
Closest in time.
The mechanistic basis of data dependence and abrupt learning in an in-context classification task
Gautam Reddy · 2024
Closest in time.
Function vectors in large language models
Eric Todd, Millicent L. Li, Arnab Sen Sharma, Aaron Mueller, Byron C. Wallace, and David Bau · 2024
Closest in time.
How many pretraining tasks are needed for in-context learning of linear regression?
Jingfeng Wu, Difan Zou, Zixiang Chen, Vladimir Braverman, Quanquan Gu, and Peter L Bartlett · 2024
Closest in time.
Can mllms perform text-to-image in-context learning?
Yuchen Zeng, Wonjun Kang, Yicong Chen, Hyung Il Koo, and Kangwook Lee · 2024
Closest in time.