Spectral Journey: How Transformers Predict the Shortest Path

Understand

Decoder-only transformers lead to a step-change in capability of large language models.

• However, opinions are mixed as to whether they are really planning or reasoning.
• A path to making progress in this direction is to study the model's behavior in a setting with carefully controlled data.
• Then interpret the learned representations and reverse-engineer the computation performed internally.