Sheaf Discovery with Joint Computation Graph Pruning and Flexible Granularity

Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer

Original

Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020 · 1910

Earlier work this paper cites.

DistilBERT, a distilled version of BERT: Smaller, faster, cheaper and lighter

Original

Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. 2020 · 1910

Earlier work this paper cites.

Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation

Original

Yoshua Bengio, Nicholas Léonard, and Aaron Courville. 2013 · 2013

Earlier work this paper cites.

Attention is All you Need

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017 · 2017

Earlier work this paper cites.

Learning Sparse Neural Networks through L_0 Regularization

Christos Louizos, Max Welling, and Diederik P. Kingma. 2018 · 2018

Earlier work this paper cites.

BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019 · 2019

Earlier work this paper cites.

The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks

Jonathan Frankle and Michael Carbin. 2019 · 2019

Earlier work this paper cites.

Language Models are Unsupervised Multitask Learners

Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019 · 2019

Earlier work this paper cites.

Zoom In: An Introduction to Circuits

Chris Olah, Nick Cammarata, Ludwig Schubert, Gabriel Goh, Michael Petrov, and Shan Carter. 2020 · 2020

Earlier work this paper cites.

Winning the Lottery with Continuous Sparsification

Pedro Savarese, Hugo Silva, and Michael Maire. 2020 · 2020

Earlier work this paper cites.

Investigating Gender Bias in Language Models Using Causal Mediation Analysis

Jesse Vig, Sebastian Gehrmann, Yonatan Belinkov, Sharon Qian, Daniel Nevo, Yaron Singer, and Stuart Shieber. 2020 · 2020

Earlier work this paper cites.

BLiMP: The Benchmark of Linguistic Minimal Pairs for English

Alex Warstadt, Alicia Parrish, Haokun Liu, Anhad Mohananey, Wei Peng, Sheng-Fu Wang, and Samuel R. Bowman. 2020 · 2020

Earlier work this paper cites.

Low-Complexity Probing via Finding Subnetworks

Steven Cao, Victor Sanh, and Alexander Rush. 2021 · 2021

Earlier work this paper cites.

Are Neural Nets Modular? Inspecting Functional Modularity Through Differentiable Weight Masks

Róbert Csordás, Sjoerd van Steenkiste, and Jürgen Schmidhuber. 2021 · 2021

Earlier work this paper cites.

Measuring and Improving Consistency in Pretrained Language Models

Yanai Elazar, Nora Kassner, Shauli Ravfogel, Abhilasha Ravichander, Eduard Hovy, Hinrich Schütze, and Yoav Goldberg. 2021 · 2021

Earlier work this paper cites.

A Mathematical Framework for Transformer Circuits

Nelson Elhage, Neel Nanda, Catherine Olsson, Tom Henighan, Nicholas Joseph, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Nova DasSarma, Dawn Drain, Deep Ganguli, Zac Hatfield-Dodds, Danny Hernandez, Andy Jones, Jackson Kernion, Liane Lovitt, Kamal Ndousse, and 6 others. 2021 · 2021

Earlier work this paper cites.

Causal Analysis of Syntactic Agreement Mechanisms in Neural Language Models

Matthew Finlayson, Aaron Mueller, Sebastian Gehrmann, Stuart Shieber, Tal Linzen, and Yonatan Belinkov. 2021 · 2021

Earlier work this paper cites.

Causal Abstractions of Neural Networks

Atticus Geiger, Hanson Lu, Thomas Icard, and Christopher Potts. 2021 · 2021

Earlier work this paper cites.

Transformer Feed-Forward Layers Are Key-Value Memories

Mor Geva, Roei Schuster, Jonathan Berant, and Omer Levy. 2021 · 2021

Cited alongside, same era.

Parameter-Efficient Transfer Learning with Diff Pruning

Demi Guo, Alexander Rush, and Yoon Kim. 2021 · 2021

Cited alongside, same era.

Can Subnetwork Structure be the Key to Out-of-Distribution Generalization?

Dinghuai Zhang, Kartik Ahuja, Yilun Xu, Yisen Wang, and Aaron Courville. 2021 · 2021

Cited alongside, same era.

Knowledge Neurons in Pretrained Transformers

Damai Dai, Li Dong, Yaru Hao, Zhifang Sui, Baobao Chang, and Furu Wei. 2022 · 2022

Cited alongside, same era.

Sparse Interventions in Language Models with Differentiable Masking

Nicola De Cao, Leon Schmid, Dieuwke Hupkes, and Ivan Titov. 2022 · 2022

Cited alongside, same era.

Transformer Feed-Forward Layers Build Predictions by Promoting Concepts in the Vocabulary Space

GPT-4 Technical Report

Original

OpenAI. 2023 · 2023

Later among the works it cites.

Llama 2: Open Foundation and Fine-Tuned Chat Models

Original

Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, and 49 others. 2023 · 2023

Later among the works it cites.

Interpretability at Scale: Identifying Causal Mechanisms in Alpaca

Zhengxuan Wu, Atticus Geiger, Thomas Icard, Christopher Potts, and Noah Goodman. 2023 · 2023

Later among the works it cites.

Towards Best Practices of Activation Patching in Language Models: Metrics and Methods

Fred Zhang and Neel Nanda. 2023 · 2023

Later among the works it cites.

Finding Transformer Circuits With Edge Pruning

Adithya Bhaskar, Alexander Wettig, Dan Friedman, and Danqi Chen. 2024 · 2024