Overcoming Sparsity Artifacts in Crosscoders to Interpret Chat-Tuning

Understand

Model diffing is the study of how fine-tuning changes a model's representations and internal algorithms.

• Many behaviors of interest are introduced during fine-tuning, and model diffing offers a promising lens to interpret such behaviors.
• Crosscoders are a recent model diffing method that learns a shared dictionary of interpretable concepts represented as latent directions in both the base and fine-tuned models, allowing us to track how concepts shift or emerge during fine-tuning.
• Notably, prior work has observed concepts with no direction in the base model, and it was hypothesized that these model-specific latents were concepts introduced during fine-tuning.