When can transformers compositionally generalize in-context?

HyperNetworks

David Ha, Andrew M. Dai, and Quoc V. Le · 2017

Earlier work this paper cites.

Exploring the limits of transfer learning with a unified text-to-text transformer

Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu · 2020

Earlier work this paper cites.

On the Compositional Generalization Gap of In-Context Learning

Arian Hosseini, Ankit Vani, Dzmitry Bahdanau, Alessandro Sordoni, and Aaron Courville · 2022

Earlier work this paper cites.

Is a modular architecture enough?

Sarthak Mittal, Yoshua Bengio, and Guillaume Lajoie · 2022

Earlier work this paper cites.

HyperTransformer: Model generation for supervised and semi-supervised few-shot learning

Andrey Zhmoginov, Mark Sandler, and Maksym Vladymyrov · 2022

Cited alongside, same era.

What learning algorithm is in-context learning? Investigations with linear models

Ekin Akyürek, Dale Schuurmans, Jacob Andreas, Tengyu Ma, and Denny Zhou · 2023

Cited alongside, same era.

How Do In-Context Examples Affect Compositional Generalization?

Shengnan An, Zeqi Lin, Qiang Fu, Bei Chen, Nanning Zheng, Jian-Guang Lou, and Dongmei Zhang · 2023

Cited alongside, same era.

Why Can GPT Learn In-Context? Language Models Secretly Perform Gradient Descent as Meta-Optimizers

Damai Dai, Yutao Sun, Li Dong, Yaru Hao, Shuming Ma, Zhifang Sui, and Furu Wei · 2023

Cited alongside, same era.

Attention as a hypernetwork

Original

Simon Schug, Seijin Kobayashi, Yassir Akram, João Sacramento, and Razvan Pascanu

Cited in the paper.

Discovering modular solutions that generalize compositionally

Simon Schug, Seijin Kobayashi, Yassir Akram, Maciej Wołczyk, Alexandra Proca, Johannes von Oswald, Razvan Pascanu, João Sacramento, and Angelika Steger

Cited in the paper.

Human-like systematic generalization through a meta-learning neural network

Brenden M. Lake and Marco Baroni · 2023

Later among the works it cites.

Uncovering mesa-optimization algorithms in Transformers, September 2023

Original

Johannes von Oswald, Eyvind Niklasson, Maximilian Schlegel, Seijin Kobayashi, Nicolas Zucchet, Nino Scherrer, Nolan Miller, Mark Sandler, Blaise Agüera y Arcas, Max Vladymyrov, Razvan Pascanu, and João Sacramento · 2023

Later among the works it cites.

Does learning the right latent variables necessarily improve in-context learning?, 2024

Sarthak Mittal, Eric Elmoznino, Leo Gagnon, Sangnie Bhardwaj, Dhanya Sridhar ∗ , and Guillaume Lajoie ∗ · 2024

Closest in time.