2023

Analyzing Leakage of Personally Identifiable Information in Language Models

Lukas, Nils, Salem, Ahmed, Sim, Robert et al.

Understand

Language Models (LMs) have been shown to leak information about training data through sentence-level membership inference and reconstruction attacks.

Understanding the risk of LMs leaking Personally Identifiable Information (PII) has received less attention, which can be attributed to the false assumption that dataset curation techniques such as scrubbing are sufficient to prevent PII leakage.
Scrubbing techniques reduce but do not prevent the risk of PII leakage: in practice scrubbing is imperfect and must balance the trade-off between minimizing disclosure and preserving the utility of the dataset.
On the other hand, it is unclear to which extent algorithmic defenses such as differential privacy, designed to guarantee sentence- or user-level privacy, prevent PII disclosure.

Built on

T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell et al. , “Language models are few-shot learners,” in Advances in neural information processing systems (NeurIPS) , vol. 33. Curran Associates, Inc., 2020, pp. 1877–1901
1901
Earlier work this paper cites.
N. Carlini, S. Chien, M. Nasr, S. Song, A. Terzis, and F. Tramer, “Membership inference attacks from first principles,” in 43rd IEEE Symposium on Security and Privacy (S&P) . IEEE, 2022, pp. 1897–1914
1914
Earlier work this paper cites.
P. Samarati and L. Sweeney, “Protecting privacy when disclosing information: k k -anonymity and its enforcement through generalization and suppression,” Computer Science Laboratory, SRI International, Tech. Rep., 1998. [Online]. Available: http://www.csl.sri.com/papers/sritr-98-04/
1998
Earlier work this paper cites.
B. Klimt and Y. Yang, “Introducing the Enron corpus,” in International Conference on Email and Anti-Spam, CEAS , vol. 45, 2004, pp. 92–96
2004
Earlier work this paper cites.
C. Dwork, K. Kenthapadi, F. McSherry, I. Mironov, and M. Naor, “Our data, ourselves: Privacy via distributed noise generation,” in Advances in Cryptology – EUROCRYPT 2006 . Springer, 2006, pp. 486–503
2006
Earlier work this paper cites.
C. Dwork, F. McSherry, K. Nissim, and A. Smith, “Calibrating noise to sensitivity in private data analysis,” in Theory of cryptography conference, TCC . Springer, 2006, pp. 265–284
2006
Earlier work this paper cites.
P. Golle, “Revisiting the uniqueness of simple demographics in the us population,” in 5th ACM Workshop on Privacy in Electronic Society . ACM, 2006, pp. 77–80
2006
Earlier work this paper cites.
E. Hovy, M. Marcus, M. Palmer, L. Ramshaw, and R. Weischedel, “OntoNotes: The 90% solution,” in Human Language Technology Conference of the NAACL . ACL, 2006, pp. 57–60
2006
Earlier work this paper cites.
D. Nadeau and S. Sekine, “A survey of named entity recognition and classification,” Lingvisticae Investigationes , vol. 30, no. 1, pp. 3–26, 2007
2007
Earlier work this paper cites.
S. Bird, E. Klein, and E. Loper, Natural language processing with Python: analyzing text with the Natural Language Toolkit . O’Reilly Media, 2009
2009
Earlier work this paper cites.
R. Sennrich, B. Haddow, and A. Birch, “Neural machine translation of rare words with subword units,” arXiv preprint arXiv:1508.07909 [cs.CL] , 2015
Original
2015
Earlier work this paper cites.
M. Abadi, A. Chu, I. Goodfellow, H. B. McMahan, I. Mironov, K. Talwar, and L. Zhang, “Deep learning with differential privacy,” in 23rd ACM SIGSAC Conference on Computer and Communications Security, CCS . ACM, 2016, pp. 308–318
2016
Earlier work this paper cites.
European Union, “Regulation (EU) 2016/679 of the European Parliament and of the Council of 27 april 2016 on the protection of natural persons with regard to the processing of personal data and on the free movement of such data, and repealing Directive 95/46/EC (General Data Protection Regulation),” Official Journal , vol. L 110, pp. 1–88, 2016
2016
Earlier work this paper cites.
G. Lample, M. Ballesteros, S. Subramanian, K. Kawakami, and C. Dyer, “Neural architectures for named entity recognition,” in Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT . ACL, 2016, pp. 260–270
2016
Earlier work this paper cites.
R. Shokri, M. Stronati, C. Song, and V. Shmatikov, “Membership inference attacks against machine learning models,” in 37th IEEE symposium on security and privacy (S&P) . IEEE, 2017, pp. 3–18
2017
Earlier work this paper cites.
A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is all you need,” Advances in neural information processing systems (NeurIPS) , vol. 30, p. 6000–6010, 2017
2017
Earlier work this paper cites.
A. Fan, M. Lewis, and Y. Dauphin, “Hierarchical neural story generation,” in 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) . ACL, 2018, pp. 889–898
2018
Earlier work this paper cites.
A. Akbik, T. Bergmann, D. Blythe, K. Rasul, S. Schweter, and R. Vollgraf, “FLAIR: An easy-to-use framework for state-of-the-art NLP,” in 2019 Conference of the North American Chapter of the ACL (demonstrations) . ACL, 2019, pp. 54–59
2019
Earlier work this paper cites.
P. Budzianowski and I. Vulić, “Hello, it’s GPT-2 – How can I help you? towards the use of pretrained language models for task-oriented dialogue systems,” arXiv preprint arXiv:1907.05774 [cs.CL] , 2019
Original
2019
Earlier work this paper cites.
N. Carlini, C. Liu, Ú. Erlingsson, J. Kos, and D. Song, “The secret sharer: Evaluating and testing unintended memorization in neural networks,” in 28th USENIX Security Symposium (USENIX Security 19) , 2019, pp. 267–284
2019
Earlier work this paper cites.
I. Chalkidis, I. Androutsopoulos, and N. Aletras, “Neural legal judgment prediction in English,” in 57th Annual Meeting of the Association for Computational Linguistics, ACL . ACL, 2019, pp. 4317–4323
2019
Earlier work this paper cites.
M. X. Chen, B. N. Lee, G. Bansal, Y. Cao, S. Zhang, J. Lu, J. Tsay, Y. Wang, A. M. Dai, Z. Chen et al. , “Gmail smart compose: Real-time assisted writing,” in 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining , 2019, pp. 2287–2295
2019
Earlier work this paper cites.
J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “BERT: Pre-training of deep bidirectional transformers for language understanding,” in Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT . ACL, 2019, pp. 4171–4186
2019
Earlier work this paper cites.

Similar

A. Hoang, A. Bosselut, A. Celikyilmaz, and Y. Choi, “Efficient adaptation of pretrained transformers for abstractive summarization,” arXiv preprint arXiv:1906.00138 [cs.CL] , 2019
Original
2019
Cited alongside, same era.
Y. Liu, M. Ott, N. Goyal, J. Du, M. Joshi, D. Chen, O. Levy, M. Lewis, L. Zettlemoyer, and V. Stoyanov, “RoBERTa: A robustly optimized BERT pretraining approach,” arXiv preprint arXiv:1907.11692 [cs.CL] , 2019
Original
2019
Cited alongside, same era.
I. Loshchilov and F. Hutter, “Decoupled weight decay regularization,” in 7th International Conference on Learning Representations, ICLR . OpenReview.net, 2019
2019
Cited alongside, same era.
N. Miao, H. Zhou, L. Mou, R. Yan, and L. Li, “CGMH: Constrained sentence generation by metropolis-hastings sampling,” in AAAI Conference on Artificial Intelligence , vol. 33, no. 01, 2019, pp. 6834–6842
2019
Cited alongside, same era.
L. Rocher, J. M. Hendrickx, and Y.-A. De Montjoye, “Estimating the success of re-identifications in incomplete datasets using generative models,” Nature communications , vol. 10, no. 1, pp. 1–9, 2019
2019
Cited alongside, same era.
V. Feldman, “Does learning require memorization? a short tale about a long tail,” in 52nd Annual ACM SIGACT Symposium on Theory of Computing . ACM, 2020, pp. 954–959
2020
Cited alongside, same era.
M. Honnibal, I. Montani, S. Van Landeghem, and A. Boyd, “spaCy: Industrial-strength natural language processing in Python,” 2020
2020
Cited alongside, same era.
T. Humphries, S. Oya, L. Tulloch, M. Rafuse, I. Goldberg, U. Hengartner, and F. Kerschbaum, “Investigating membership inference attacks under data dependencies,” arXiv preprint arXiv:2010.12112 [cs.CR] , 2020
Original
2020
Cited alongside, same era.
C. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y. Zhou, W. Li, P. J. Liu et al. , “Exploring the limits of transfer learning with a unified text-to-text transformer.” J. Mach. Learn. Res. , vol. 21, no. 140, pp. 1–67, 2020
2020
Cited alongside, same era.
S. Yeom, I. Giacomelli, A. Menaged, M. Fredrikson, and S. Jha, “Overfitting, robustness, and malicious algorithms: A study of potential causes of privacy risk in machine learning,” Journal of Computer Security , vol. 28, no. 1, pp. 35–70, 2020
2020
Cited alongside, same era.
S. Zanella-Béguelin, L. Wutschitz, S. Tople, V. Rühle, A. Paverd, O. Ohrimenko, B. Köpf, and M. Brockschmidt, “Analyzing information leakage of updates to natural language models,” in 27th ACM SIGSAC Conference on Computer and Communications Security , 2020, pp. 363–375
2020
Cited alongside, same era.
N. Carlini, F. Tramer, E. Wallace, M. Jagielski, A. Herbert-Voss, K. Lee, A. Roberts, T. Brown, D. Song, U. Erlingsson et al. , “Extracting training data from large language models,” in 30th USENIX Security Symposium . USENIX Association, 2021, pp. 2633–2650
2021
Cited alongside, same era.

Then

W. Fedus, B. Zoph, and N. Shazeer, “Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity,” Journal of Machine Learning Research , vol. 23, no. 120, pp. 1–39, 2022
2022
Later among the works it cites.
J. Hoffmann, S. Borgeaud, A. Mensch, E. Buchatskaya, T. Cai, E. Rutherford, D. d. L. Casas, L. A. Hendricks, J. Welbl, A. Clark, T. Hennigan, E. Noland, K. Millican, G. v. d. Driessche, B. Damoc, A. Guy, S. Osindero, K. Simonyan, E. Elsen, J. W. Rae, O. Vinyals, and L. Sifre, “Training compute-optimal large language models,” arXiv preprint arXiv:2203.15556 [cs.LG] , 2022
Original
2022
Later among the works it cites.
D. Ippolito, F. Tramèr, M. Nasr, C. Zhang, M. Jagielski, K. Lee, C. A. Choquette-Choo, and N. Carlini, “Preventing verbatim memorization in language models gives a false sense of privacy,” arXiv preprint arXiv:2210.17546 [cs.LG] , 2022
Original
2022
Later among the works it cites.
M. Jagielski, O. Thakkar, F. Tramèr, D. Ippolito, K. Lee, N. Carlini, E. Wallace, S. Song, A. Thakurta, N. Papernot et al. , “Measuring forgetting of memorized training examples,” arXiv preprint arXiv:2207.00099 [cs.LG] , 2022
Original
2022
Later among the works it cites.
B. Jayaraman, E. Ghosh, M. Chase, S. Roy, H. Inan, W. Dai, and D. Evans, “Combing for credentials: Active pattern extraction from smart reply,” arXiv preprint arXiv:2203.07618 [cs.CR] , 2022
Original
2022
Later among the works it cites.
N. Kandpal, E. Wallace, and C. Raffel, “Deduplicating training data mitigates privacy risks in language models,” in 39th International Conference on Machine Learning , ser. Proceedings of Machine Learning Research, vol. 162. PMLR, 2022, pp. 10 697–10 707
2022
Later among the works it cites.
J. Lee, T. Le, J. Chen, and D. Lee, “Do language models plagiarize?” arXiv preprint arXiv:2203.07618 [cs.CL] , 2022
Original
2022
Later among the works it cites.
X. Li, F. Tramer, P. Liang, and T. Hashimoto, “Large language models can be strong differentially private learners,” in 10th International Conference on Learning Representations (ICLR) . OpenReview.net, 2022
2022
Later among the works it cites.
B. McMahan and A. Thakurta, “Federated learning with formal differential privacy guarantees,” Google, Tech. Rep., 2022. [Online]. Available: https://ai.googleblog.com/2022/02/federated-learning-with-formal.html
2022
Later among the works it cites.
F. Mireshghallah, K. Goyal, A. Uniyal, T. Berg-Kirkpatrick, and R. Shokri, “Quantifying privacy risks of masked language models using membership inference attacks,” arXiv preprint arXiv:2203.03929 [cs.LG] , 2022
Original
2022
Later among the works it cites.
F. Mireshghallah, A. Uniyal, T. Wang, D. Evans, and T. Berg-Kirkpatrick, “Memorization in NLP fine-tuning methods,” arXiv preprint arXiv:2205.12506 [cs.CL] , 2022
Original
2022
Later among the works it cites.
R. Parikh, C. Dupuy, and R. Gupta, “Canary extraction in natural language understanding models,” in 60th Annual Meeting of the Association for Computational Linguistics, ACL . ACL, 2022, pp. 552–560
2022
Later among the works it cites.
I. Pilán, P. Lison, L. Øvrelid, A. Papadopoulou, D. Sánchez, and M. Batet, “The text anonymization benchmark (TAB): A dedicated corpus and evaluation framework for text anonymization,” Computational Linguistics , vol. 48, no. 4, pp. 1053–1101, 2022
2022
Later among the works it cites.
W. Shi, A. Cui, E. Li, R. Jia, and Z. Yu, “Selective differential privacy for language modeling,” in Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NACACL-HLT . ACL, 2022, pp. 2848–2859
2022
Later among the works it cites.
P. Stock, I. Shilov, I. Mironov, and A. Sablayrolles, “Defending against reconstruction attacks with Rényi differential privacy,” arXiv preprint arXiv:2202.07623 [cs.LG] , 2022
Original
2022
Later among the works it cites.
K. Tirumala, A. H. Markosyan, L. Zettlemoyer, and A. Aghajanyan, “Memorization without overfitting: Analyzing the training dynamics of large language models,” in Advances in neural information processing systems (NeurIPS) , vol. 35. Curran Associates, Inc., 2022, pp. 38 274–38 290
2022
Later among the works it cites.
T. Vakili, A. Lamproudis, A. Henriksson, and H. Dalianis, “Downstream task performance of BERT models pre-trained using automatically de-identified clinical data,” in 13th Language Resources and Evaluation Conference, LREC . ACL, 2022, pp. 4245–4252
2022
Later among the works it cites.
D. Yu, S. Naik, A. Backurs, S. Gopi, H. A. Inan, G. Kamath, J. Kulkarni, Y. T. Lee, A. Manoel, L. Wutschitz, S. Yekhanin, and H. Zhang, “Differentially private fine-tuning of language models,” in 10th International Conference on Learning Representations (ICLR) . OpenReview.net, 2022
2022
Later among the works it cites.
R. Zhang, S. Hidano, and F. Koushanfar, “Text revealer: Private text reconstruction via model inversion attacks against transformers,” arXiv preprint arXiv:2209.10505 [cs.CL] , 2022
Original
2022
Later among the works it cites.
X. Zhao, L. Li, and Y.-X. Wang, “Provably confidential language modelling,” in Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NACACL-HLT . ACL, 2022, pp. 943–955
2022
Later among the works it cites.
N. Lukas, A. Salem, R. Sim, S. Tople, L. Wutschitz, and S. Zanella-Béguelin, “Analyzing leakage of personally identifiable information in language models,” in 2023 IEEE Symposium on Security and Privacy (SP) . IEEE, 2023
2023
Closest in time.
J. Huang, H. Shao, and K. C.-C. Chang, “Are large pre-trained language models leaking your personal information?” in Findings of the Association for Computational Linguistics, EMNLP . ACL, 2022, pp. 2038–2047
2047
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…