Context-Aware Differential Privacy for Language Modeling

Exploring the limits of transfer learning with a unified text-to-text transformer, 2019

Original

C. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y. Zhou, W. Li, and P. J. Liu · 1910

Earlier work this paper cites.

Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter, 2019

Original

V. Sanh, L. Debut, J. Chaumond, and T. Wolf · 1910

Earlier work this paper cites.

Training production language models without memorizing user data, 2020

Original

S. Ramaswamy, O. Thakkar, R. Mathews, G. Andrew, H. B. McMahan, and F. Beaufays · 2009

Earlier work this paper cites.

The algorithmic foundations of differential privacy

C. Dwork, A. Roth, et al · 2014

Earlier work this paper cites.

Deep learning with differential privacy

Abadi and et al · 2016

Earlier work this paper cites.

Pointer sentinel mixture models

Original

S. Merity, C. Xiong, J. Bradbury, and R. Socher · 2016

Earlier work this paper cites.

Differential Privacy Has Disparate Impact on Model Accuracy

E. Bagdasaryan, O. Poursaeed, and V. Shmatikov · 2019

Cited alongside, same era.

The secret sharer: Evaluating and testing unintended memorization in neural networks

N. Carlini, C. Liu, U. Erlingsson, J. Kos, and D. Song · 2019

Cited alongside, same era.

Language models are unsupervised multitask learners

A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, and I. Sutskever · 2019

Cited alongside, same era.

Language models are few-shot learners

T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, S. Agarwal, A. Herbert-Voss, G. Krueger, T. Henighan, R. Child, A. Ramesh, D. Ziegler, J. Wu, C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, S. Gray, B. Chess, J. Clark, C. Berner, S. McCandlish, A. Radford, I. Sutskever, and D. Amodei · 2020

Cited alongside, same era.

Does learning require memorization? a short tale about a long tail

V. Feldman · 2020

Cited alongside, same era.

Extracting training data from large language models

N. Carlini, F. Tramèr, E. Wallace, M. Jagielski, A. Herbert-Voss, K. Lee, A. Roberts, T. Brown, D. Song, Ú. Erlingsson, A. Oprea, and C. Raffel · 2021

Later among the works it cites.

Training data leakage analysis in language models, 2021

Original

H. A. Inan, O. Ramadan, L. Wutschitz, D. Jones, V. Rühle, J. Withers, and R. Sim · 2021

Later among the works it cites.

How can we know when language models know? on the calibration of language models for question answering

Z. Jiang, J. Araki, H. Ding, and G. Neubig · 2021

Later among the works it cites.

What does it mean for a language model to preserve privacy?

H. Brown, K. Lee, F. Mireshghallah, R. Shokri, and F. Tramèr · 2022

Later among the works it cites.

Selective differential privacy for language modeling

W. Shi, A. Cui, E. Li, R. Jia, and Z. Yu · 2022

Later among the works it cites.