Automated Interpretability Metrics Do Not Distinguish Trained and Random Transformers

Understand

Sparse autoencoders (SAEs) are widely used to extract sparse, interpretable latents from transformer activations.

• We test whether commonly used SAE quality metrics and automatic explanation pipelines can distinguish trained transformers from randomly initialized ones (e.g., where parameters are sampled i.i.d.
• from a Gaussian).
• Over a wide range of Pythia model sizes and multiple randomization schemes, we find that, in many settings, SAEs trained on randomly initialized transformers produce auto-interpretability scores and reconstruction metrics that are similar to those from trained models.