Gender Biases and Where to Find Them: Exploring Gender Bias in Pre-Trained Transformer-based Language Models Using Movement Pruning

Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter

Original

Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. 2019 · 1910

Earlier work this paper cites.

Huggingface’s transformers: State-of-the-art natural language processing

Original

Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, and Jamie Brew. 2019 · 1910

Earlier work this paper cites.

Assessing the benchmarking capacity of machine reading comprehension datasets

Original

Saku Sugawara, Pontus Stenetorp, Kentaro Inui, and Akiko Aizawa. 2020 · 1911

Earlier work this paper cites.

Movement pruning: Adaptive sparsity by fine-tuning

Original

Victor Sanh, Thomas Wolf, and Alexander M. Rush. 2020 · 2005

Earlier work this paper cites.

Learning both weights and connections for efficient neural network

Original

Song Han, Jeff Pool, John Tran, and William J. Dally. 2015 · 2015

Earlier work this paper cites.

Semantics derived automatically from language corpora contain human-like biases

Aylin Caliskan, Joanna J. Bryson, and Arvind Narayanan. 2017 · 2017

Earlier work this paper cites.

Attention is all you need

Original

Ashish Vaswani, Noam M. Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017 · 2017

Earlier work this paper cites.

Mixed precision training

Original

Paulius Micikevicius, Sharan Narang, Jonah Alben, Gregory Frederick Diamos, Erich Elsen, David García, Boris Ginsburg, Michael Houston, Oleksii Kuchaiev, Ganesh Venkatesh, and Hao Wu. 2018 · 2018

Earlier work this paper cites.

GLUE: A multi-task benchmark and analysis platform for natural language understanding

Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2018 · 2018

Earlier work this paper cites.

BERT: Pre-training of deep bidirectional transformers for language understanding

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019 · 2019

Cited alongside, same era.

Gender-preserving debiasing for pre-trained word embeddings

Masahiro Kaneko and Danushka Bollegala. 2019 · 2019

Cited alongside, same era.

On measuring social biases in sentence encoders

Chandler May, Alex Wang, Shikha Bordia, Samuel R. Bowman, and Rachel Rudinger. 2019 · 2019

Cited alongside, same era.

Analyzing multi-head self-attention: Specialized heads do the heavy lifting, the rest can be pruned

Elena Voita, David Talbot, Fedor Moiseev, Rico Sennrich, and Ivan Titov. 2019 · 2019

Cited alongside, same era.

Neural network acceptability judgments

Alex Warstadt, Amanpreet Singh, and Samuel R. Bowman. 2019 · 2019

Cited alongside, same era.

Gender bias in contextualized word embeddings

Jieyu Zhao, Tianlu Wang, Mark Yatskar, Ryan Cotterell, Vicente Ordonez, and Kai-Wei Chang. 2019 · 2019

Gender bias in neural natural language processing

Original

Kaiji Lu, Piotr Mardziel, Fangjing Wu, Preetam Amancharla, and Anupam Datta. 2020 · 2020

Later among the works it cites.

PowerTransformer: Unsupervised controllable revision for biased language correction

Xinyao Ma, Maarten Sap, Hannah Rashkin, and Yejin Choi. 2020 · 2020

Later among the works it cites.

Automatically neutralizing subjective bias in text

Reid Pryzant, Richard Diehl Martinez, Nathan Dass, Sadao Kurohashi, Dan Jurafsky, and Diyi Yang. 2020 · 2020

Later among the works it cites.

Debiasing pre-trained contextualised embeddings

Masahiro Kaneko and Danushka Bollegala. 2021 · 2021

Later among the works it cites.

Block pruning for faster transformers

François Lagunas, Ella Charlaix, Victor Sanh, and Alexander Rush. 2021 · 2021

Later among the works it cites.

StereoSet: Measuring stereotypical bias in pretrained language models