Transformer Circuit Faithfulness Metrics are not Robust

A survey of methods for explaining black box models

Riccardo Guidotti, Anna Monreale, Salvatore Ruggieri, Franco Turini, Fosca Giannotti, and Dino Pedreschi · 2018

Earlier work this paper cites.

Analyzing and interpreting neural networks for nlp: A report on the first blackboxnlp workshop

Afra Alishahi, Grzegorz Chrupała, and Tal Linzen · 2019

Earlier work this paper cites.

Curve circuits

Nick Cammarata, Gabriel Goh, Shan Carter, Chelsea Voss, Ludwig Schubert, and Chris Olah · 2020

Earlier work this paper cites.

Towards faithfully interpretable NLP systems: How should we define and evaluate faithfulness?

Alon Jacovi and Yoav Goldberg · 2020

Earlier work this paper cites.

Zoom in: An introduction to circuits

Chris Olah, Nick Cammarata, Ludwig Schubert, Gabriel Goh, Michael Petrov, and Shan Carter · 2020

Earlier work this paper cites.

Investigating gender bias in language models using causal mediation analysis

Jesse Vig, Sebastian Gehrmann, Yonatan Belinkov, Sharon Qian, Daniel Nevo, Yaron Singer, and Stuart Shieber · 2020

Earlier work this paper cites.

A mathematical framework for transformer circuits

Nelson Elhage, Neel Nanda, Catherine Olsson, Tom Henighan, Nicholas Joseph, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Nova DasSarma, Dawn Drain, Deep Ganguli, Zac Hatfield-Dodds, Danny Hernandez, Andy Jones, Jackson Kernion, Liane Lovitt, Kamal Ndousse, Dario Amodei, Tom Brown, Jack Clark, Jared Kaplan, Sam McCandlish, and Chris Olah · 2021

Earlier work this paper cites.

Causal scrubbing: a method for rigorously testing interpretability hypotheses [redwood research], 2022

Lawrence Chan, Adrià Garriga-Alonso, Nicholas Goldowsky-Dill, Ryan Greenblatt, Jenny Nitishinskaya, Ansh Radhakrishnan, Buck Shlegeris, and Nate Thomas · 2022

Earlier work this paper cites.

Locating and editing factual associations in GPT

Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov · 2022

Earlier work this paper cites.

Transformerlens

Neel Nanda and Joseph Bloom · 2022

Earlier work this paper cites.

In-context learning and induction heads

Catherine Olsson, Nelson Elhage, Neel Nanda, Nicholas Joseph, Nova DasSarma, Tom Henighan, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Dawn Drain, Deep Ganguli, Zac Hatfield-Dodds, Danny Hernandez, Scott Johnston, Andy Jones, Jackson Kernion, Liane Lovitt, Kamal Ndousse, Dario Amodei, Tom Brown, Jack Clark, Jared Kaplan, Sam McCandlish, and Chris Olah · 2022

Earlier work this paper cites.

Towards best practices of activation patching in language models: Metrics and methods, 2024

Fred Zhang and Neel Nanda · 2022

Earlier work this paper cites.

Pythia: A suite for analyzing large language models across training and scaling, 2023

Stella Biderman, Hailey Schoelkopf, Quentin Anthony, Herbie Bradley, Kyle O’Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, Aviya Skowron, Lintang Sutawika, and Oskar van der Wal · 2023

Earlier work this paper cites.

Language models can explain neurons in language models

Steven Bills, Nick Cammarata, Dan Mossing, Henk Tillman, Leo Gao, Gabriel Goh, Ilya Sutskever, Jan Leike, Jeff Wu, and William Saunders · 2023

Cited alongside, same era.

Towards monosemanticity: Decomposing language models with dictionary learning

Trenton Bricken, Adly Templeton, Joshua Batson, Brian Chen, Adam Jermyn, Tom Conerly, Nick Turner, Cem Anil, Carson Denison, Amanda Askell, Robert Lasenby, Yifan Wu, Shauna Kravec, Nicholas Schiefer, Tim Maxwell, Nicholas Joseph, Zac Hatfield-Dodds, Alex Tamkin, Karina Nguyen, Brayden McLean, Josiah E Burke, Tristan Hume, Shan Carter, Tom Henighan, and Christopher Olah · 2023

Cited alongside, same era.

A toy model of universality: Reverse engineering how networks learn group operations, 2023

Bilal Chughtai, Lawrence Chan, and Neel Nanda · 2023

Cited alongside, same era.

Towards automated circuit discovery for mechanistic interpretability

Arthur Conmy, Augustine N. Mavor-Parker, Aengus Lynch, Stefan Heimersheim, and Adrià Garriga-Alonso · 2023

Cited alongside, same era.

Sparse autoencoders find highly interpretable features in language models, 2023

Hoagy Cunningham, Aidan Ewart, Logan Riggs, Robert Huben, and Lee Sharkey · 2023

Attribution patching outperforms automated circuit discovery, 9 2023

Aaquib Syed and Can Rager · 2023

Later among the works it cites.

Linear representations of sentiment in large language models, 2023

Curt Tigges, Oskar John Hollinsworth, Atticus Geiger, and Neel Nanda · 2023

Later among the works it cites.

Activation addition: Steering language models without optimization, 2023

Alexander Matt Turner, Lisa Thiergart, David Udell, Gavin Leech, Ulisse Mini, and Monte MacDiarmid · 2023

Later among the works it cites.

Interpretability in the wild: a circuit for indirect object identification in GPT-2 small

Kevin Ro Wang, Alexandre Variengien, Arthur Conmy, Buck Shlegeris, and Jacob Steinhardt · 2023

Later among the works it cites.

Faithfulness vs. plausibility: On the (un)reliability of explanations from large language models, 2024

Original

Chirag Agarwal, Sree Harsha Tanneru, and Himabindu Lakkaraju · 2024

Closest in time.