In-Context Learning of Linear Dynamical Systems with Transformers: Approximation Bounds and Depth-Separation

Understand

This paper investigates approximation-theoretic aspects of the in-context learning capability of the transformers in representing a family of noisy linear dynamical systems.

• Our first theoretical result establishes an upper bound on the approximation error of multi-layer transformers with respect to an $L^2$-testing loss uniformly defined across tasks.
• This result demonstrates that transformers with logarithmic depth can achieve error bounds comparable with those of the least-squares estimator.
• In contrast, our second result establishes a non-diminishing lower bound on the approximation error for a class of single-layer linear transformers, which suggests a depth-separation phenomenon for transformers in the in-context learning of dynamical systems.