Inner Loop Inference for Pretrained Transformers: Unlocking Latent Capabilities Without Training

Understand

Deep Learning architectures, and in particular Transformers, are conventionally viewed as a composition of layers.

• These layers are actually often obtained as the sum of two contributions: a residual path that copies the input and the output of a Transformer block.
• As a consequence, the inner representations (i.e.
• the input of these blocks) can be interpreted as iterative refinement of a propagated latent representation.