Fetching the paper…
Reading the bibliography…
Sparse autoencoders (SAEs) are a recent technique for decomposing neural network activations into human-interpretable features.
Adaptive mixtures of local experts
Robert A Jacobs, Michael I Jordan, Steven J Nowlan, and Geoffrey E Hinton · 1991
Earlier work this paper cites.
Hierarchical mixtures of experts and the em algorithm
Michael I Jordan and Robert A Jacobs · 1994
Earlier work this paper cites.
Sparse coding with an overcomplete basis set: A strategy employed by v1?
Bruno A Olshausen and David J Field · 1997
Earlier work this paper cites.
Method of optimal directions for frame design
Kjersti Engan, Sven Ole Aase, and J Hakon Husoy · 1999
Earlier work this paper cites.
Mixtures of gaussian processes
Volker Tresp · 2000
Earlier work this paper cites.
A parallel mixture of svms for very large scale problems
Ronan Collobert, Samy Bengio, and Yoshua Bengio · 2001
Earlier work this paper cites.
Infinite mixtures of gaussian process experts
Carl Rasmussen and Zoubin Ghahramani · 2001
Earlier work this paper cites.
K-svd: An algorithm for designing overcomplete dictionaries for sparse representation
Michal Aharon, Michael Elad, and Alfred Bruckstein · 2006
Earlier work this paper cites.
Sparse deep belief net model for visual area v2
Honglak Lee, Chaitanya Ekanadham, and Andrew Ng · 2007
Earlier work this paper cites.
Large-scale deep unsupervised learning using graphics processors
Rajat Raina, Anand Madhavan, and Andrew Y Ng · 2009
Earlier work this paper cites.
Sparse and redundant representations: from theory to applications in signal and image processing
Michael Elad · 2010
Earlier work this paper cites.
Building high-level features using large scale unsupervised learning
Quoc V Le · 2013
Earlier work this paper cites.
Alireza Makhzani and Brendan Frey · 2013
Earlier work this paper cites.
Adam: A method for stochastic optimization
Diederik P Kingma · 2014
Earlier work this paper cites.
Zero-bias autoencoders and the benefits of co-adapting features
Kishore Konda, Roland Memisevic, and David Krueger · 2014
Earlier work this paper cites.
Expert gate: Lifelong learning with a network of experts
Rahaf Aljundi, Punarjay Chakravarty, and Tinne Tuytelaars · 2017
Earlier work this paper cites.
In-datacenter performance analysis of a tensor processing unit
Norman P Jouppi, Cliff Young, Nishant Patil, David Patterson, Gaurav Agrawal, Raminder Bajwa, Sarah Bates, Suresh Bhatia, Nan Boden, Al Borchers, et al · 2017
Earlier work this paper cites.
Outrageously large neural networks: The sparsely-gated mixture-of-experts layer
Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean · 2017
Cited alongside, same era.
Openwebtext corpus
Aaron Gokaslan and Vanya Cohen · 2019
Cited alongside, same era.
Language models are unsupervised multitask learners
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al · 2019
Cited alongside, same era.
Language Models are Few-Shot Learners
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei · 2020
Cited alongside, same era.
Slide: In defense of smart algorithms over hardware acceleration for large-scale deep learning systems
Beidi Chen, Tharun Medini, James Farwell, Charlie Tai, Anshumali Shrivastava, et al · 2020
Transformer circuits april 2024 update: Update on how we train saes, 2024b
Transformer Circuits Team Anthropic · 2024
Closest in time.
Transformer circuits august 2024 update: Evals case study, 2024c
Transformer Circuits Team Anthropic · 2024
Closest in time.
Identifying functionally important features with end-to-end sparse dictionary learning
Dan Braun, Jordan Taylor, Nicholas Goldowsky-Dill, and Lee Sharkey · 2024
Closest in time.
BatchTopK: A simple improvement for TopK-SAEs
Bart Bussmann, Patrick Leask, and Neel Nanda · 2024
Closest in time.
Deepseekmoe: Towards ultimate expert specialization in mixture-of-experts language models
Damai Dai, Chengqi Deng, Chenggang Zhao, RX Xu, Huazuo Gao, Deli Chen, Jiashi Li, Wangding Zeng, Xingkai Yu, Y Wu, et al · 2024
Closest in time.
alphaXiv searches the wider corpus for related work and actual follow-ups.
alphaXiv is searching for related work…
Cited alongside, same era.
Reformer: The efficient transformer
Nikita Kitaev, Łukasz Kaiser, and Anselm Levskaya · 2020
Cited alongside, same era.
Gshard: Scaling giant models with conditional computation and automatic sharding
Dmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu, Dehao Chen, Orhan Firat, Yanping Huang, Maxim Krikun, Noam Shazeer, and Zhifeng Chen · 2020
Cited alongside, same era.
Zoom in: An introduction to circuits
Chris Olah, Nick Cammarata, Ludwig Schubert, Gabriel Goh, Michael Petrov, and Shan Carter · 2020
Cited alongside, same era.
Flashattention: Fast and memory-efficient exact attention with io-awareness
Tri Dao, Dan Fu, Stefano Ermon, Atri Rudra, and Christopher Ré · 2022
Cited alongside, same era.
Toy models of superposition
Nelson Elhage, Tristan Hume, Catherine Olsson, Nicholas Schiefer, Tom Henighan, Shauna Kravec, Zac Hatfield-Dodds, Robert Lasenby, Dawn Drain, Carol Chen, Roger Grosse, Sam McCandlish, Jared Kaplan, Dario Amodei, Martin Wattenberg, and Christopher Olah · 2022
Cited alongside, same era.
Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity
William Fedus, Barret Zoph, and Noam Shazeer · 2022
Cited alongside, same era.
Mixture-of-experts with expert choice routing
Yanqi Zhou, Tao Lei, Hanxiao Liu, Nan Du, Yanping Huang, Vincent Zhao, Andrew M Dai, Quoc V Le, James Laudon, et al · 2022
Cited alongside, same era.
Joshua Engels, Isaac Liao, Eric J Michaud, Wes Gurnee, and Max Tegmark · 2024
Closest in time.
Scaling and evaluating sparse autoencoders, 2024
Leo Gao, Tom Dupré la Tour, Henk Tillman, Gabriel Goh, Rajan Troll, Alec Radford, Ilya Sutskever, Jan Leike, and Jeffrey Wu · 2024
Closest in time.
Albert Q Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, et al · 2024
Closest in time.
Open source automated interpretability for sparse autoencoder features
Caden Juang, Gonçalo Paulo, Jacob Drori, and Nora Belrose · 2024
Closest in time.
The remarkable robustness of llms: Stages of inference?
Vedang Lad, Wes Gurnee, and Max Tegmark · 2024
Closest in time.
dictionary learning
Samuel Marks, Adam Karvonen, and Aaron Mueller · 2024
Closest in time.
ProLU: A nonlinearity for sparse autoencoders
Glen Taggart · 2024
Closest in time.
Gemma 2: Improving open language models at a practical size
Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupatiraju, Léonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ramé, et al · 2024
Closest in time.
Scaling monosemanticity: Extracting interpretable features from claude 3 sonnet
Adly Templeton, Tom Conerly, Jonathan Marcus, Jack Lindsey, Trenton Bricken, Brian Chen, Adam Pearce, Craig Citro, Emmanuel Ameisen, Andy Jones, Hoagy Cunningham, Nicholas L Turner, Callum McDougall, Monte MacDiarmid, C. Daniel Freeman, Theodore R. Sumers, Edward Rees, Joshua Batson, Adam Jermyn, Shan Carter, Chris Olah, and Tom Henighan · 2024
Closest in time.
Addressing feature suppression in SAEs
Benjamin Wright and Lee Sharkey · 2024
Closest in time.
Grok-1 model card, 2024
xAI · 2024
Closest in time.