2022

Truth Serum: Poisoning Machine Learning Models to Reveal Their Secrets

Tramèr, Florian, Shokri, Reza, Joaquin, Ayrton San et al.

Understand

We introduce a new class of attacks on machine learning models.

We show that an adversary who can poison a training dataset can cause models trained on this dataset to leak significant private details of training points belonging to other parties.
Our active inference attacks connect two independent lines of work targeting the integrity and privacy of machine learning training data.
Our attacks are effective across membership inference, attribute inference, and data extraction.

Built on

Protocols for secure computations
Andrew C Yao · 1982
Earlier work this paper cites.
How to play any mental game, or a completeness theorem for protocols with honest majority
Oded Goldreich, Silvio Micali, and Avi Wigderson · 1987
Earlier work this paper cites.
UCI machine learning repository: Adult data set
Ronny Kohavi and Barry Becker · 1996
Earlier work this paper cites.
Chosen ciphertext attacks against protocols based on the RSA encryption standard PKCS#1
Daniel Bleichenbacher · 1998
Earlier work this paper cites.
Compression and information leakage of plaintext
John Kelsey · 2002
Earlier work this paper cites.
Security flaws induced by CBC padding—applications to SSL, IPSEC, WTLS…
Serge Vaudenay · 2002
Earlier work this paper cites.
Calibrating noise to sensitivity in private data analysis
Cynthia Dwork, Frank McSherry, Kobbi Nissim, and Adam Smith · 2006
Earlier work this paper cites.
Resolving individuals contributing trace amounts of dna to highly complex mixtures using high-density snp genotyping microarrays
Nils Homer, Szabolcs Szelinger, Margot Redman, David Duggan, Waibhav Tembe, Jill Muehling, John V Pearson, Dietrich A Stephan, Stanley F Nelson, and David W Craig · 2008
Earlier work this paper cites.
Learning multiple layers of features from tiny images, 2009
Alex Krizhevsky and Geoffrey Hinton · 2009
Earlier work this paper cites.
Protecting browsers from cross-origin CSS attacks
Lin-Shung Huang, Zack Weinberg, Chris Evans, and Collin Jackson · 2010
Earlier work this paper cites.
Poisoning attacks against support vector machines
Battista Biggio, Blaine Nelson, and Pavel Laskov · 2012
Earlier work this paper cites.
Breach: reviving the crime attack
Yoel Gluck, Neal Harris, and Angelo Prado · 2013
Earlier work this paper cites.
Privacy in pharmacogenetics: An end-to-end case study of personalized warfarin dosing
Matthew Fredrikson, Eric Lantz, Somesh Jha, Simon Lin, David Page, and Thomas Ristenpart · 2014
Earlier work this paper cites.
Model inversion attacks that exploit confidence information and basic countermeasures
Matt Fredrikson, Somesh Jha, and Thomas Ristenpart · 2015
Earlier work this paper cites.
Deep learning with differential privacy
Martin Abadi, Andy Chu, Ian Goodfellow, H. Brendan McMahan, Ilya Mironov, Kunal Talwar, and Li Zhang · 2016
Earlier work this paper cites.
Google’s neural machine translation system: Bridging the gap between human and machine translation
Original
Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V. Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, Jeff Klingner, Apurva Shah, Melvin Johnson, Xiaobing Liu, Lukasz Kaiser, Stephan Gouws, Yoshikiyo Kato, Taku Kudo, Hideto Kazawa, Keith Stevens, George Kurian, Nishant Patil, Wei Wang, Cliff Young, Jason Smith, Jason Riesa, Alex Rudnick, Oriol Vinyals, Greg Corrado, Macduff Hughes, and Jeffrey Dean · 2016
Earlier work this paper cites.
Wide residual networks
Sergey Zagoruyko and Nikos Komodakis · 2016
Earlier work this paper cites.
Privacy-preserving deep learning via additively homomorphic encryption
Yoshinori Aono, Takuya Hayashi, Lihua Wang, and Shiho Moriai · 2017
Earlier work this paper cites.
Learning from untrusted data
Moses Charikar, Jacob Steinhardt, and Gregory Valiant · 2017
Earlier work this paper cites.
Deep models under the GAN: Information leakage from collaborative deep learning
Briland Hitaj, Giuseppe Ateniese, and Fernando Perez-Cruz · 2017
Earlier work this paper cites.
Neural trojans
Yuntao Liu, Yang Xie, and Ankur Srivastava · 2017
Earlier work this paper cites.
Pointer sentinel mixture models
Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher · 2017
Earlier work this paper cites.
SecureML: A system for scalable privacy-preserving machine learning
Payman Mohassel and Yupeng Zhang · 2017
Earlier work this paper cites.
Towards poisoning of deep learning algorithms with back-gradient optimization
Luis Muñoz-González, Battista Biggio, Ambra Demontis, Andrea Paudice, Vasin Wongrassamee, Emil C Lupu, and Fabio Roli · 2017
Earlier work this paper cites.
Membership inference attacks against machine learning models
Reza Shokri, Marco Stronati, Congzheng Song, and Vitaly Shmatikov · 2017
Earlier work this paper cites.
Machine learning models that remember too much
Congzheng Song, Thomas Ristenpart, and Vitaly Shmatikov · 2017
Earlier work this paper cites.

Similar

Manipulating machine learning: Poisoning attacks and countermeasures for regression learning
Matthew Jagielski, Alina Oprea, Battista Biggio, Chang Liu, Cristina Nita-Rotaru, and Bo Li · 2018
Cited alongside, same era.
Trojaning attack on neural networks
Yingqi Liu, Shiqing Ma, Yousra Aafer, Wen-Chuan Lee, Juan Zhai, Weihang Wang, and Xiangyu Zhang · 2018
Cited alongside, same era.
Towards deep learning models resistant to adversarial attacks
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu · 2018
Cited alongside, same era.
ABY3: A mixed protocol framework for machine learning
Payman Mohassel and Peter Rindal · 2018
Cited alongside, same era.
Machine learning with membership privacy using adversarial regularization
Milad Nasr, Reza Shokri, and Amir Houmansadr · 2018
Cited alongside, same era.
Poison frogs! Targeted clean-label poisoning attacks on neural networks
Ali Shafahi, W Ronny Huang, Mahyar Najibi, Octavian Suciu, Christoph Studer, Tudor Dumitras, and Tom Goldstein · 2018
Cited alongside, same era.
When does machine learning FAIL? Generalized transferability for evasion and poisoning attacks
Octavian Suciu, Radu Marginean, Yigitcan Kaya, Hal Daume III, and Tudor Dumitras · 2018
Cited alongside, same era.
Spectral signatures in backdoor attacks
Brandon Tran, Jerry Li, and Aleksander Madry · 2018
Cited alongside, same era.
Privacy risk in machine learning: Analyzing the connection to overfitting
Samuel Yeom, Irene Giacomelli, Matt Fredrikson, and Somesh Jha · 2018
Cited alongside, same era.
Analyzing federated learning through an adversarial lens
Arjun Nitin Bhagoji, Supriyo Chakraborty, Prateek Mittal, and Seraphin Calo · 2019
Cited alongside, same era.
The secret sharer: Evaluating and testing unintended memorization in neural networks
Nicholas Carlini, Chang Liu, Úlfar Erlingsson, Jernej Kos, and Dawn Song · 2019
Cited alongside, same era.
Sever: A robust meta-algorithm for stochastic optimization
Ilias Diakonikolas, Gautam Kamath, Daniel Kane, Jerry Li, Jacob Steinhardt, and Alistair Stewart · 2019
Cited alongside, same era.

Then

A pragmatic approach to membership inferences on machine learning models
Yunhui Long, Lei Wang, Diyue Bu, Vincent Bindschaedler, Xiaofeng Wang, Haixu Tang, Carl A Gunter, and Kai Chen · 2020
Later among the works it cites.
Training production language models without memorizing user data
Original
Swaroop Ramaswamy, Om Thakkar, Rajiv Mathews, Galen Andrew, H Brendan McMahan, and Françoise Beaufays · 2020
Later among the works it cites.
Analyzing information leakage of updates to natural language models
Santiago Zanella-Béguelin, Lukas Wutschitz, Shruti Tople, Victor Rühle, Andrew Paverd, Olga Ohrimenko, Boris Köpf, and Marc Brockschmidt · 2020
Later among the works it cites.
Blind backdoors in deep learning models
Eugene Bagdasaryan and Vitaly Shmatikov · 2021
Later among the works it cites.
On the dangers of stochastic parrots: Can language models be too big?
Emily M Bender, Timnit Gebru, Angelina McMillan-Major, and Shmargaret Shmitchell · 2021
Later among the works it cites.
When the curious abandon honesty: Federated learning is not private
Original
Franziska Boenisch, Adam Dziedzic, Roei Schuster, Ali Shahin Shamsabadi, Ilia Shumailov, and Nicolas Papernot · 2021
Later among the works it cites.
Extracting training data from large language models
Nicholas Carlini, Florian Tramer, Eric Wallace, Matthew Jagielski, Ariel Herbert-Voss, Katherine Lee, Adam Roberts, Tom Brown, Dawn Song, Ulfar Erlingsson, et al · 2021
Later among the works it cites.
Adversarial examples make strong poisons
Liam Fowl, Micah Goldblum, Ping-yeh Chiang, Jonas Geiping, Wojciech Czaja, and Tom Goldstein · 2021
Later among the works it cites.
Witches’ brew: Industrial scale data poisoning via gradient matching
Jonas Geiping, Liam H Fowl, W. Ronny Huang, Wojciech Czaja, Gavin Taylor, Michael Moeller, and Tom Goldstein · 2021
Later among the works it cites.
Revisiting membership inference under realistic assumptions
Bargav Jayaraman, Lingxiao Wang, David Evans, and Quanquan Gu · 2021
Later among the works it cites.
Advances and open problems in federated learning
Peter Kairouz, H Brendan McMahan, Brendan Avent, Aurélien Bellet, Mehdi Bennis, Arjun Nitin Bhagoji, Kallista Bonawitz, Zachary Charles, Graham Cormode, Rachel Cummings, et al · 2021
Later among the works it cites.
Antipodes of label differential privacy: PATE and ALIBI
Mani Malek Esmaeili, Ilya Mironov, Karthik Prasad, Igor Shilov, and Florian Tramer · 2021
Later among the works it cites.
Adversary instantiation: Lower bounds for differentially private machine learning
Milad Nasr, Shuang Songi, Abhradeep Thakurta, Nicolas Papemoti, and Nicholas Carlin · 2021
Later among the works it cites.
Learning transferable visual models from natural language supervision
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al · 2021
Later among the works it cites.
Unadversarial examples: Designing objects for robust vision
Hadi Salman, Andrew Ilyas, Logan Engstrom, Sai Vemprala, Aleksander Madry, and Ashish Kapoor · 2021
Later among the works it cites.
You autocomplete me: Poisoning vulnerabilities in neural code completion
Roei Schuster, Congzheng Song, Eran Tromer, and Vitaly Shmatikov · 2021
Later among the works it cites.
Understanding unintended memorization in language models under federated learning
Om Dipakbhai Thakkar, Swaroop Ramaswamy, Rajiv Mathews, and Francoise Beaufays · 2021
Later among the works it cites.
Membership inference attacks from first principles
Nicholas Carlini, Steve Chien, Milad Nasr, Shuang Song, Andreas Terzis, and Florian Tramer · 2022
Closest in time.
Quantifying memorization across neural language models
Original
Nicholas Carlini, Daphne Ippolito, Matthew Jagielski, Katherine Lee, Florian Tramer, and Chiyuan Zhang · 2022
Closest in time.
Decepticons: Corrupted transformers breach privacy in federated learning for language models
Original
Liam Fowl, Jonas Geiping, Steven Reich, Yuxin Wen, Wojtek Czaja, Micah Goldblum, and Tom Goldstein · 2022
Closest in time.
Property inference from poisoning
S. Mahloujifar, E. Ghosh, and M. Chase · 2022
Closest in time.
Are your sensitive attributes private? Novel model inversion attribute inference attacks on classification models
Shagufta Mehnaz, Sayanton V Dibbo, Ehsanul Kabir, Ninghui Li, and Elisa Bertino · 2022
Closest in time.
Quantifying privacy risks of masked language models using membership inference attacks
Original
Fatemehsadat Mireshghallah, Kartik Goyal, Archit Uniyal, Taylor Berg-Kirkpatrick, and Reza Shokri · 2022
Closest in time.
On the importance of difficulty calibration in membership inference attacks
Lauren Watson, Chuan Guo, Graham Cormode, and Alexandre Sablayrolles · 2022
Closest in time.
Fishing for user data in large-batch federated learning via gradient magnification
Yuxin Wen, Jonas A. Geiping, Liam Fowl, Micah Goldblum, and Tom Goldstein · 2022
Closest in time.
Enhanced membership inference attacks against machine learning models
Jiayuan Ye, Aadyaa Maddi, Sasi Kumar Murakonda, and Reza Shokri · 2022
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…