Causal Distillation for Language Models

TinyBERT: Distilling BERT for natural language understanding

Original

Xiaoqi Jiao, Yichun Yin, Lifeng Shang, Xin Jiang, Xiao Chen, Linlin Li, Fang Wang, and Qun Liu. 2019 · 1909

Earlier work this paper cites.

DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter

Original

Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. 2019 · 1910

Earlier work this paper cites.

Causation, Prediction, and Search , 2nd edition

Peter Spirtes, Clark N Glymour, and Richard Scheines. 2001 · 2001

Earlier work this paper cites.

Introduction to the CoNLL-2003 shared task: Language-independent named entity recognition

Erik F. Tjong Kim Sang and Fien De Meulder. 2003 · 2003

Earlier work this paper cites.

Causal mediation analysis for interpreting neural NLP: The case of gender bias

Original

Jesse Vig, Sebastian Gehrmann, Yonatan Belinkov, Sharon Qian, Daniel Nevo, Yaron Singer, and Stuart Shieber. 2020 · 2004

Earlier work this paper cites.

Causality: Models, Reasoning and Inference , 2nd edition

Judea Pearl. 2009 · 2009

Earlier work this paper cites.

Distilling the knowledge in a neural network

Original

Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. 2015 · 2015

Earlier work this paper cites.

Pointer sentinel mixture models

Original

Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. 2016 · 2016

Cited alongside, same era.

SQuAD: 100,000+ questions for machine comprehension of text

Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016 · 2016

Cited alongside, same era.

GLUE: A multi-task benchmark and analysis platform for natural language understanding

Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2018 · 2018

Cited alongside, same era.

Abstracting causal models

Sander Beckers and Joseph Y. Halpern. 2019 · 2019

Cited alongside, same era.

BERT: Pre-training of deep bidirectional transformers for language understanding

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019 · 2019

Cited alongside, same era.

Neural natural language inference models partially embed theories of lexical entailment and negation

Atticus Geiger, Kyle Richardson, and Christopher Potts. 2020 · 2020

Later among the works it cites.

Analysing lexical semantic change with contextualised word representations

Mario Giulianelli, Marco Del Tredici, and Raquel Fernández. 2020 · 2020

Later among the works it cites.

Transformers: State-of-the-art natural language processing

Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. 2020 · 2020

Later among the works it cites.

Amnesic probing: Behavioral explanation with amnesic counterfactuals

Yanai Elazar, Shauli Ravfogel, Alon Jacovi, and Yoav Goldberg. 2021 · 2021

Closest in time.

CausaLM: Causal Model Explanation Through Counterfactual Language Models

Amir Feder, Nadav Oved, Uri Shalit, and Roi Reichart. 2021 · 2021

Closest in time.