Investigating Representation Universality: Case Study on Genealogical Representations

Understand

Motivated by interpretability and reliability, we investigate whether large language models (LLMs) deploy universal geometric structures to encode discrete, graph-structured knowledge.

• To this end, we present two complementary experimental evidence that might support universality of graph representations.
• First, on an in-context genealogy Q&A task, we train a cone probe to isolate a tree-like subspace in residual stream activations and use activation patching to verify its causal effect in answering related questions.
• We validate our findings across five different models.