Decoding Dark Matter: Specialized Sparse Autoencoders for Interpreting Rare Concepts in Foundation Models

Distributionally robust neural networks for group shifts: On the importance of regularization for worst-case generalization

Original

Shiori Sagawa, Pang Wei Koh, Tatsunori B Hashimoto, and Percy Liang. 2019 · 1911

Earlier work this paper cites.

Convergent and discriminant validation by the multitrait-multimethod matrix

Donald T Campbell and Donald W Fiske. 1959 · 1959

Earlier work this paper cites.

Matching pursuits with time-frequency dictionaries

Stéphane G Mallat and Zhifeng Zhang. 1993 · 1993

Earlier work this paper cites.

Emergence of simple-cell receptive field properties by learning a sparse code for natural images

Bruno A Olshausen and David J Field. 1996 · 1996

Earlier work this paper cites.

Sparse coding with an overcomplete basis set: A strategy employed by v1?

Bruno A Olshausen and David J Field. 1997 · 1997

Earlier work this paper cites.

Causal mediation analysis for interpreting neural nlp: The case of gender bias

Original

Jesse Vig, Sebastian Gehrmann, Yonatan Belinkov, Sharon Qian, Daniel Nevo, Simas Sakenis, Jason Huang, Yaron Singer, and Stuart Shieber. 2020 · 2004

Earlier work this paper cites.

Compressed sensing

David L Donoho. 2006 · 2006

Earlier work this paper cites.

Reducing the dimensionality of data with neural networks

Geoffrey E Hinton and Ruslan R Salakhutdinov. 2006 · 2006

Earlier work this paper cites.

Sparse deep belief net model for visual area v2

Honglak Lee, Chaitanya Ekanadham, and Andrew Ng. 2007 · 2007

Earlier work this paper cites.

Tilted empirical risk minimization

Original

Tian Li, Ahmad Beirami, Maziar Sanjabi, and Virginia Smith. 2020 · 2007

Earlier work this paper cites.

The probabilistic relevance framework: Bm25 and beyond

Stephen Robertson and Hugo Zaragoza. 2009 · 2009

Earlier work this paper cites.

Deep learning of representations: Looking forward

Yoshua Bengio. 2013 · 2013

Earlier work this paper cites.

Building high-level features using large scale unsupervised learning

Quoc V Le. 2013 · 2013

Earlier work this paper cites.

Zero-bias autoencoders and the benefits of co-adapting features

Original

Kishore Konda, Roland Memisevic, and David Krueger. 2014 · 2014

Earlier work this paper cites.

Sparse modeling for image and vision processing

Julien Mairal, Francis Bach, Jean Ponce, et al. 2014 · 2014

Earlier work this paper cites.

Adam: A method for stochastic optimization

Original

Diederik P Kingma and Jimmy Ba. 2015 · 2015

Earlier work this paper cites.

Decoupled weight decay regularization

Original

I Loshchilov. 2017 · 2017

Earlier work this paper cites.

A characterization of guesswork on swiftly tilting curves

Ahmad Beirami, Robert Calderbank, Mark M Christiansen, Ken R Duffy, and Muriel Médard. 2018 · 2018

Earlier work this paper cites.

Isolating sources of disentanglement in variational autoencoders

Ricky TQ Chen, Xuechen Li, Roger B Grosse, and David K Duvenaud. 2018 · 2018

Earlier work this paper cites.

Interpretability beyond feature attribution: Quantitative testing with concept activation vectors (tcav)

Been Kim, Martin Wattenberg, Justin Gilmer, Carrie Cai, James Wexler, Fernanda Viegas, et al. 2018 · 2018

Earlier work this paper cites.

Disentangling by factorising

Hyunjik Kim and Andriy Mnih. 2018 · 2018

Earlier work this paper cites.

Confounding variables can degrade generalization performance of radiological deep learning models

Original

John R Zech, Marcus A Badgeley, Manway Liu, Anthony B Costa, Joseph J Titano, and Eric K Oermann. 2018 · 2018

Earlier work this paper cites.

Bias in bios: A case study of semantic representation bias in a high-stakes setting

Maria De-Arteaga, Alexey Romanov, Hanna Wallach, Jennifer Chayes, Christian Borgs, Alexandra Chouldechova, Sahin Geyik, Krishnaram Kenthapadi, and Adam Tauman Kalai. 2019 · 2019

Earlier work this paper cites.

Disentangling disentanglement in variational autoencoders

Emile Mathieu, Tom Rainforth, Nana Siddharth, and Yee Whye Teh. 2019 · 2019

Earlier work this paper cites.

The pile: An 800gb dataset of diverse text for language modeling

Original

Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, et al. 2020 · 2020

Earlier work this paper cites.

Zoom in: An introduction to circuits

Chris Olah, Nick Cammarata, Ludwig Schubert, Gabriel Goh, Michael Petrov, and Shan Carter. 2020 · 2020

Earlier work this paper cites.

Estimating training data influence by tracing gradient descent

Garima Pruthi, Frederick Liu, Satyen Kale, and Mukund Sundararajan. 2020 · 2020

Earlier work this paper cites.

On the opportunities and risks of foundation models

Original

Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, et al. 2021 · 2021

Cited alongside, same era.

A mathematical framework for transformer circuits

Nelson Elhage, Neel Nanda, Catherine Olsson, Tom Henighan, Nicholas Joseph, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, et al. 2021 · 2021

Cited alongside, same era.

Towards a theoretical framework of out-of-distribution generalization

Haotian Ye, Chuanlong Xie, Tianle Cai, Ruichen Li, Zhenguo Li, and Liwei Wang. 2021 · 2021

Cited alongside, same era.

Transformer visualization via dictionary learning: contextualized embedding as a linear superposition of transformer factors

Original

Zeyu Yun, Yubei Chen, Bruno A Olshausen, and Yann LeCun. 2021 · 2021

Cited alongside, same era.

Probing classifiers: Promises, shortcomings, and advances

Yonatan Belinkov. 2022 · 2022

Representation engineering: A top-down approach to ai transparency

Original

Andy Zou, Long Phan, Sarah Chen, James Campbell, Phillip Guo, Richard Ren, Alexander Pan, Xuwang Yin, Mantas Mazeika, Ann-Kathrin Dombrowski, et al. 2023 · 2023

Later among the works it cites.

Examining language model performance with reconstructed activations using sparse autoencoders

Evan Anders and Joseph Bloom. 2024 · 2024

Closest in time.

arxiv physics dataset

Anonymous. 2024 · 2024

Closest in time.

Claude 3.5 sonnet

Anthropic. 2024 · 2024

Closest in time.

Gemma-2b-residual-stream-saes

John Bloom. 2024 · 2024

Closest in time.

Identifying functionally important features with end-to-end sparse dictionary learning

Original

Dan Braun, Jordan Taylor, Nicholas Goldowsky-Dill, and Lee Sharkey. 2024 · 2024