2024

Patchscopes: A Unifying Framework for Inspecting Hidden Representations of Language Models

Ghandeharioun, Asma, Caciularu, Avi, Pearce, Adam et al.

Understand

Understanding the internal representations of large language models (LLMs) can help explain models' behavior and verify their alignment with human values.

Given the capabilities of LLMs in generating human-understandable text, we propose leveraging the model itself to explain its internal representations in natural language.
We introduce a framework called Patchscopes and show how it can be used to answer a wide range of questions about an LLM's computation.
We show that many prior interpretability methods based on projecting representations into the vocabulary space and intervening on the LLM computation can be viewed as instances of this framework.

Built on

Sentence-bert: Sentence embeddings using siamese bert-networks
Original
Reimers, N. and Gurevych, I · 1908
Earlier work this paper cites.
ROUGE: A package for automatic evaluation of summaries
Lin, C.-Y · 2004
Earlier work this paper cites.
Distributional vectors encode referential attributes
Gupta, A., Boleda, G., Baroni, M., and Padó, S · 2015
Earlier work this paper cites.
What’s in an embedding? analyzing word embeddings through multilingual evaluation
Köhn, A · 2015
Earlier work this paper cites.
Understanding image representations by measuring their equivariance and equivalence
Lenc, K. and Vedaldi, A · 2015
Earlier work this paper cites.
Pointer sentinel mixture models
Merity, S., Xiong, C., Bradbury, J., and Socher, R · 2016
Earlier work this paper cites.
Understanding intermediate layers using linear classifier probes
Alain, G. and Bengio, Y · 2017
Earlier work this paper cites.
Lstmvis: A tool for visual analysis of hidden state dynamics in recurrent neural networks
Strobelt, H., Gehrmann, S., Pfister, H., and Rush, A. M · 2017
Earlier work this paper cites.
Interpretability beyond feature attribution: Quantitative testing with concept activation vectors (tcav)
Kim, B., Wattenberg, M., Gilmer, J., Cai, C., Wexler, J., Viegas, F., et al · 2018
Earlier work this paper cites.
Interpreting deep visual representations via network dissection
Zhou, B., Bau, D., Oliva, A., and Torralba, A · 2018
Earlier work this paper cites.
Analysis methods in neural language processing: A survey
Belinkov, Y. and Glass, J · 2019
Earlier work this paper cites.
Bert rediscovers the classical nlp pipeline
Tenney, I., Das, D., and Pavlick, E · 2019
Earlier work this paper cites.
The Pile: An 800GB dataset of diverse text for language modeling
Original
Gao, L., Biderman, S., Black, S., Golding, L., Hoppe, T., Foster, C., Phang, J., He, H., Thite, A., Nabeshima, N., et al · 2020
Earlier work this paper cites.
interpreting gpt: the logit lens
nostalgebraist · 2020
Earlier work this paper cites.
Investigating gender bias in language models using causal mediation analysis
Vig, J., Gehrmann, S., Belinkov, Y., Qian, S., Nevo, D., Singer, Y., and Shieber, S · 2020
Earlier work this paper cites.
BERTnesia: Investigating the capture and forgetting of knowledge in BERT
Wallat, J., Singh, J., and Anand, A · 2020
Earlier work this paper cites.
Revisiting model stitching to compare neural representations
Bansal, Y., Nakkiran, P., and Barak, B · 2021
Earlier work this paper cites.
Similarity and matching of neural network representations
Csiszárik, A., Kőrösi-Szabó, P., Matszangosz, Á. K., Papp, G., and Varga, D · 2021
Earlier work this paper cites.
Causal abstractions of neural networks
Geiger, A., Lu, H., Icard, T., and Potts, C · 2021
Earlier work this paper cites.
Dissect: Disentangled simultaneous explanations via concept traversals
Original
Ghandeharioun, A., Kim, B., Li, C.-L., Jou, B., Eoff, B., and Picard, R. W · 2021
Earlier work this paper cites.
Mapping language models to grounded conceptual spaces
Patel, R. and Pavlick, E · 2021
Earlier work this paper cites.
GPT-J-6B: A 6 Billion Parameter Autoregressive Language Model
Wang, B. and Komatsuzaki, A · 2021
Earlier work this paper cites.

Similar

Probing classifiers: Promises, shortcomings, and advances
Belinkov, Y · 2022
Cited alongside, same era.
Sok: Toward transparent ai: A survey on interpreting the inner structures of deep neural networks
Casper, S., Rauker, T., Ho, A., and Hadfield-Menell, D · 2022
Cited alongside, same era.
LM-debugger: An interactive tool for inspection and intervention in transformer-based language models
Geva, M., Caciularu, A., Dar, G., Roit, P., Sadde, S., Shlain, M., Tamir, B., and Goldberg, Y · 2022
Cited alongside, same era.
Transformer feed-forward layers build predictions by promoting concepts in the vocabulary space
Geva, M., Caciularu, A., Wang, K., and Goldberg, Y · 2022
Cited alongside, same era.
Natural language descriptions of deep features
Hernandez, E., Schwettmann, S., Bau, D., Bagashvili, T., Torralba, A., and Andreas, J · 2022
Cited alongside, same era.
Post-hoc interpretability for neural nlp: A survey
Madsen, A., Reddy, S., and Chandar, S · 2022
Cited alongside, same era.
Linearly mapping from image to text space
Merullo, J., Castricato, L., Eickhoff, C., and Pavlick, E · 2022
Cited alongside, same era.
Interpretability in the wild: a circuit for indirect object identification in GPT-2 small
Wang, K. R., Variengien, A., Conmy, A., Shlegeris, B., and Steinhardt, J · 2022
Cited alongside, same era.
Chain-of-thought prompting elicits reasoning in large language models
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Xia, F., Chi, E., Le, Q. V., Zhou, D., et al · 2022
Cited alongside, same era.
Eliciting latent predictions from transformers with the tuned lens
Original
Belrose, N., Furman, Z., Smith, L., Halawi, D., Ostrovsky, I., McKinney, L., Biderman, S., and Steinhardt, J · 2023
Cited alongside, same era.
Pythia: A suite for analyzing large language models across training and scaling
Biderman, S., Schoelkopf, H., Anthony, Q. G., Bradley, H., O’Brien, K., Hallahan, E., Khan, M. A., Purohit, S., Prashanth, U. S., Raff, E., et al · 2023
Cited alongside, same era.
Language models can explain neurons in language models
Bills, S., Cammarata, N., Mossing, D., Tillman, H., Gao, L., Goh, G., Sutskever, I., Leike, J., Wu, J., and Saunders, W · 2023
Cited alongside, same era.

Then

Decoderlens: Layerwise interpretation of encoder-decoder transformers
Original
Langedijk, A., Mohebbi, H., Sarti, G., Zuidema, W., and Jumelet, J · 2023
Later among the works it cites.
Does circuit analysis interpretability scale? evidence from multiple choice capabilities in chinchilla
Original
Lieberum, T., Rahtz, M., Kramár, J., Irving, G., Shah, R., and Mikulik, V · 2023
Later among the works it cites.
Self-refine: Iterative refinement with self-feedback
Madaan, A., Tandon, N., Gupta, P., Hallinan, S., Gao, L., Wiegreffe, S., Alon, U., Dziri, N., Prabhumoye, S., Yang, Y., Gupta, S., Majumder, B. P., Hermann, K., Welleck, S., Yazdanbakhsh, A., and Clark, P · 2023
Later among the works it cites.
When not to trust language models: Investigating effectiveness of parametric and non-parametric memories
Mallen, A., Asai, A., Zhong, V., Das, R., Khashabi, D., and Hajishirzi, H · 2023
Later among the works it cites.
A mechanism for solving relational tasks in transformer language models
Original
Merullo, J., Eickhoff, C., and Pavlick, E · 2023
Later among the works it cites.
Can llms facilitate interpretation of pre-trained language models?
Original
Mousi, B., Durrani, N., and Dalvi, F · 2023
Later among the works it cites.
Emergent linear representations in world models of self-supervised sequence models
Nanda, N., Lee, A., and Wattenberg, M · 2023
Later among the works it cites.
Future lens: Anticipating subsequent tokens from a single hidden state
Pal, K., Sun, J., Yuan, A., Wallace, B. C., and Bau, D · 2023
Later among the works it cites.
Multimodal neurons in pretrained text-only transformers
Schwettmann, S., Chowdhury, N., Klein, S., Bau, D., and Torralba, A · 2023
Later among the works it cites.
Explaining black box text modules in natural language with language models
Singh, C., Hsu, A., Antonello, R., Jain, S., Huth, A., Yu, B., and Gao, J · 2023
Later among the works it cites.
The curious case of hallucinatory (un)answerability: Finding truths in the hidden states of over-confident large language models
Slobodkin, A., Goldman, O., Caciularu, A., Dagan, I., and Ravfogel, S · 2023
Later among the works it cites.
A mechanistic interpretation of arithmetic reasoning in language models using causal mediation analysis
Stolfo, A., Belinkov, Y., and Sachan, M · 2023
Later among the works it cites.
Analyzing vision transformers for image classification in class embedding space
Original
Vilas, M. G., Schaumlöffel, T., and Roig, G · 2023
Later among the works it cites.
Gaussian Process Probes (GPP) for uncertainty-aware probing
Original
Wang, Z., Ku, A., Baldridge, J., Griffiths, T. L., and Kim, B · 2023
Later among the works it cites.
Tree of thoughts: Deliberate problem solving with large language models
Yao, S., Yu, D., Zhao, J., Shafran, I., Griffiths, T. L., Cao, Y., and Narasimhan, K. R · 2023
Later among the works it cites.
Give me the facts! a survey on factual knowledge probing in pre-trained language models
Youssef, P., Koraş, O., Li, M., Schlötterer, J., and Seifert, C · 2023
Later among the works it cites.
Towards best practices of activation patching in language models: Metrics and methods
Original
Zhang, F. and Nanda, N · 2023
Later among the works it cites.
Mquake: Assessing knowledge editing in language models via multi-hop questions
Original
Zhong, Z., Wu, Z., Manning, C. D., Potts, C., and Chen, D · 2023
Later among the works it cites.
Graph of thoughts: Solving elaborate problems with large language models
Besta, M., Blach, N., Kubicek, A., Gerstenberger, R., Gianinazzi, L., Gajda, J., Lehmann, T., Podstawski, M., Niewiadomski, H., Nyczyk, P., et al · 2024
Closest in time.
Understanding and patching compositional reasoning in llms
Original
Li, Z., Jiang, G., Xie, H., Song, L., Lian, D., and Wei, Y · 2024
Closest in time.
The expressive power of transformers with chain of thought
Merrill, W. and Sabharwal, A · 2024
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…