2019

Eternal Sunshine of the Spotless Net: Selective Forgetting in Deep Networks

Golatkar, Aditya, Achille, Alessandro, Soatto, Stefano

Understand

We explore the problem of selectively forgetting a particular subset of the data used for training a deep neural network.

While the effects of the data to be forgotten can be hidden from the output of the network, insights may still be gleaned by probing deep into its weights.
We propose a method for "scrubbing'" the weights clean of information about a particular set of training data.
The method does not require retraining from scratch, nor access to the data originally used for training.

Built on

Understanding deep learning requires rethinking generalization
Original
Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals · 1911
Earlier work this paper cites.
Optimal brain damage
Yann LeCun, John S Denker, and Sara A Solla · 1990
Earlier work this paper cites.
Learning multiple layers of features from tiny images
Alex Krizhevsky et al · 2009
Earlier work this paper cites.
A pac-bayesian tutorial with a d ropout bound
Original
David McAllester · 2013
Earlier work this paper cites.
The algorithmic foundations of differential privacy
Cynthia Dwork, Aaron Roth, et al · 2014
Earlier work this paper cites.
New insights and perspectives on the natural gradient method
Original
James Martens · 2014
Earlier work this paper cites.
Striving for simplicity: The all convolutional net
Original
Jost Tobias Springenberg, Alexey Dosovitskiy, Thomas Brox, and Martin Riedmiller · 2014
Earlier work this paper cites.
Towards making systems forget with machine unlearning
Yinzhi Cao and Junfeng Yang · 2015
Earlier work this paper cites.
Model inversion attacks that exploit confidence information and basic countermeasures
Matt Fredrikson, Somesh Jha, and Thomas Ristenpart · 2015
Earlier work this paper cites.
Train faster, generalize better: Stability of stochastic gradient descent
Original
Moritz Hardt, Benjamin Recht, and Yoram Singer · 2015
Earlier work this paper cites.

Similar

Variational dropout and the local reparameterization trick
Durk P Kingma, Tim Salimans, and Max Welling · 2015
Cited alongside, same era.
Optimizing neural networks with kronecker-factored approximate curvature
James Martens and Roger Grosse · 2015
Cited alongside, same era.
Deep learning with differential privacy
Martin Abadi, Andy Chu, Ian Goodfellow, H Brendan McMahan, Ilya Mironov, Kunal Talwar, and Li Zhang · 2016
Cited alongside, same era.
A closer look at memorization in deep networks
Devansh Arpit, Stanisław Jastrzębski, Nicolas Ballas, David Krueger, Emmanuel Bengio, Maxinder S Kanwal, Tegan Maharaj, Asja Fischer, Aaron Courville, Yoshua Bengio, et al · 2017
Cited alongside, same era.
Computing nonvacuous generalization bounds for deep (stochastic) neural networks with many more parameters than training data
Original
Gintare Karolina Dziugaite and Daniel M Roy · 2017
Cited alongside, same era.
Deep models under the gan: information leakage from collaborative deep learning
Briland Hitaj, Giuseppe Ateniese, and Fernando Perez-Cruz · 2017
Cited alongside, same era.
Overcoming catastrophic forgetting in neural networks
James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska, et al · 2017
Cited alongside, same era.
Knock knock, who’s there? membership inference on aggregate location data
Original
Apostolos Pyrgelis, Carmela Troncoso, and Emiliano De Cristofaro · 2017
Cited alongside, same era.
Machine learning models that remember too much
Congzheng Song, Thomas Ristenpart, and Vitaly Shmatikov · 2017
Cited alongside, same era.
Vggface2: A dataset for recognising faces across pose and age
Q. Cao, L. Shen, W. Xie, O. M. Parkhi, and A. Zisserman · 2018
Cited alongside, same era.

Then

Critical learning periods in deep neural networks
Alessandro Achille, Matteo Rovere, and Stefano Soatto · 2019
Closest in time.
Where is the Information in a Deep Neural Network?
Original
Alessandro Achille and Stefano Soatto · 2019
Closest in time.
Machine unlearning
Original
Lucas Bourtoule, Varun Chandrasekaran, Christopher Choquette-Choo, Hengrui Jia, Adelin Travers, Baiwu Zhang, David Lie, and Nicolas Papernot · 2019
Closest in time.
Making ai forget you: Data deletion in machine learning
Antonio Ginart, Melody Guan, Gregory Valiant, and James Y Zou · 2019
Closest in time.
Time matters in regularizing deep networks: Weight decay and data augmentation affect early learning dynamics, matter little near convergence
Aditya Sharad Golatkar, Alessandro Achille, and Stefano Soatto · 2019
Closest in time.
Logan: Membership inference attacks against generative models
Jamie Hayes, Luca Melis, George Danezis, and Emiliano De Cristofaro · 2019
Closest in time.
Zero-shot knowledge transfer via adversarial belief matching
Original
Paul Micaelli and Amos Storkey · 2019
Closest in time.
Verifying that the influence of a user data point has been removed from a machine learning classifier, Mar. 2019
Saurabh Shintre and Jasjeet Dhaliwal · 2019
Closest in time.
Demystifying membership inference attacks in machine learning as a service
Stacey Truex, Ling Liu, Mehmet Emre Gursoy, Lei Yu, and Wenqi Wei · 2019
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…