Towards Understanding the Relationship between In-context Learning and Compositional Generalization

Understand

According to the principle of compositional generalization, the meaning of a complex expression can be understood as a function of the meaning of its parts and of how they are combined.

• This principle is crucial for human language processing and also, arguably, for NLP models in the face of out-of-distribution data.
• However, many neural network models, including Transformers, have been shown to struggle with compositional generalization.
• In this paper, we hypothesize that forcing models to in-context learn can provide an inductive bias to promote compositional generalization.