An explainable transformer circuit for compositional generalization

Understand

Compositional generalization-the systematic combination of known components into novel structures-remains a core challenge in cognitive science and machine learning.

• Although transformer-based large language models can exhibit strong performance on certain compositional tasks, the underlying mechanisms driving these abilities remain opaque, calling into question their interpretability.
• In this work, we identify and mechanistically interpret the circuit responsible for compositional induction in a compact transformer.
• Using causal ablations, we validate the circuit and formalize its operation using a program-like description.