Differentially Private Language Models Benefit from Public Pre-training

Roberta: A robustly optimized bert pretraining approach

Original

Y. Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, M. Lewis, L. Zettlemoyer, and V. Stoyanov. 2019 · 1907

Earlier work this paper cites.

Brown corpus manual

W. N. Francis and H. Kucera. 1979 · 1979

Earlier work this paper cites.

Long short-term memory

Sepp Hochreiter and Jürgen Schmidhuber. 1997 · 1997

Earlier work this paper cites.

Train large, then compress: Rethinking model size for efficient training and inference of transformers

Original

Zhuohan Li, Eric Wallace, Sheng Shen, Kevin Lin, Kurt Keutzer, Dan Klein, and Joseph E. Gonzalez. 2020 · 2002

Earlier work this paper cites.

The algorithmic foundations of differential privacy

Cynthia Dwork and Aaron Roth. 2014 · 2014

Earlier work this paper cites.

May 2015 reddit comments

Reddit. 2019 · 2015

Cited alongside, same era.

Deep Learning with Differential Privacy

Original

Martín Abadi, Andy Chu, Ian Goodfellow, H. Brendan McMahan, Ilya Mironov, Kunal Talwar, and Li Zhang. 2016 · 2016

Cited alongside, same era.

Learning differentially private language models without losing accuracy

Original

H. Brendan McMahan, Daniel Ramage, Kunal Talwar, and Li Zhang. 2017 · 2017

Cited alongside, same era.

Fundamentals of recurrent neural network (RNN) and long short-term memory (LSTM) network

Original

Alex Sherstinsky. 2018 · 2018

Cited alongside, same era.

Bert: Pre-training of deep bidirectional transformers for language understanding

J. Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019 · 2019

Cited alongside, same era.

Generalised differential privacy for text document processing

Natasha Fernandes, Mark Dras, and Annabelle McIver. 2019 · 2019

Later among the works it cites.

Pyvacy: Privacy algorithms for pytorch

Chris Waites. 2019 · 2019

Later among the works it cites.

Improving differentially private models with active learning

Original

Zhengli Zhao, Nicolas Papernot, Sameer Singh, Neoklis Polyzotis, and Augustus Odena. 2019 · 2019

Later among the works it cites.

Privacy-and utility-preserving textual analysis via calibrated multivariate perturbations

Oluwaseyi Feyisetan, Borja Balle, Thomas Drake, and Tom Diethe. 2020 · 2020

Closest in time.