2022

Taming Normalizing Flows

Malnick, Shimon, Avidan, Shai, Fried, Ohad

Understand

We propose an algorithm for taming Normalizing Flow models - changing the probability that the model will produce a specific image or image category.

We focus on Normalizing Flows because they can calculate the exact generation probability likelihood for a given image.
We demonstrate taming using models that generate human faces, a subdomain with many interesting privacy and bias considerations.
Our method can be used in the context of privacy, e.g., removing a specific person from the output of a model, and also in the context of debiasing by forcing a model to output specific image categories according to a given target distribution.

Built on

Certified Data Removal from Machine Learning Models, Aug. 2020
Original
Chuan Guo, Tom Goldstein, Awni Hannun, and Laurens van der Maaten · 1911
Earlier work this paper cites.
On information and sufficiency
Solomon Kullback and Richard A Leibler · 1951
Earlier work this paper cites.
The Kolmogorov-Smirnov test for goodness of fit
F. J. Massey · 1951
Earlier work this paper cites.
Maximum likelihood: an introduction
Lucien Le Cam · 1990
Earlier work this paper cites.
Backpropagation and stochastic gradient descent method
Shun-ichi Amari · 1993
Earlier work this paper cites.
The influence of the sigmoid function parameters on the speed of backpropagation learning
Jun Han and Claudio Moraga · 1995
Earlier work this paper cites.
Nonlinear independent component analysis using ensemble learning: Experiments and discussion
Harri Valpola, Xavier Giannakopoulos, Antti Honkela, and Juha Karhunen · 2000
Earlier work this paper cites.
The foundations of cost-sensitive learning
Charles Elkan · 2001
Earlier work this paper cites.
Discriminative learning under covariate shift
Steffen Bickel, Michael Brückner, and Tobias Scheffer · 2009
Earlier work this paper cites.
Rectified linear units improve restricted boltzmann machines
Vinod Nair and Geoffrey E Hinton · 2010
Earlier work this paper cites.
An empirical investigation of catastrophic forgetting in gradient-based neural networks
Original
Ian J Goodfellow, Mehdi Mirza, Da Xiao, Aaron Courville, and Yoshua Bengio · 2013
Earlier work this paper cites.
Deep generative stochastic networks trainable by backprop
Yoshua Bengio, Eric Laufer, Guillaume Alain, and Jason Yosinski · 2014
Earlier work this paper cites.
Generative adversarial nets
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio · 2014
Earlier work this paper cites.
Stochastic gradient vb and the variational auto-encoder
Diederik P Kingma and Max Welling · 2014
Earlier work this paper cites.
NICE: Non-linear Independent Components Estimation, Apr. 2015
Original
Laurent Dinh, David Krueger, and Yoshua Bengio · 2015
Earlier work this paper cites.
Adam: A method for stochastic optimization
Diederik P. Kingma and Jimmy Ba · 2015
Earlier work this paper cites.
Deep learning face attributes in the wild
Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang · 2015
Earlier work this paper cites.
Variational inference with normalizing flows
Danilo Rezende and Shakir Mohamed · 2015
Earlier work this paper cites.
Variational inference with normalizing flows
Danilo Rezende and Shakir Mohamed · 2015
Earlier work this paper cites.
A note on the evaluation of generative models
Original
Lucas Theis, Aäron van den Oord, and Matthias Bethge · 2015
Earlier work this paper cites.

Similar

Deep residual learning for image recognition
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun · 2016
Cited alongside, same era.
Pixel rnn
Aäron van den Oord and Nal Kalchbrenner · 2016
Cited alongside, same era.
Density estimation using Real NVP, Feb. 2017
Original
Laurent Dinh, Jascha Sohl-Dickstein, and Samy Bengio · 2017
Cited alongside, same era.
Gans trained by a two time-scale update rule converge to a local nash equilibrium
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter · 2017
Cited alongside, same era.
Progressive growing of gans for improved quality, stability, and variation
Original
Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen · 2017
Cited alongside, same era.
Overcoming catastrophic forgetting in neural networks
Original
James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A. Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska, Demis Hassabis, Claudia Clopath, Dharshan Kumaran, and Raia Hadsell · 2017
Cited alongside, same era.
Masked autoregressive flow for density estimation
George Papamakarios, Theo Pavlakou, and Iain Murray · 2017
Cited alongside, same era.
Glow: Generative Flow with Invertible 1x1 Convolutions, July 2018
Original
Diederik P. Kingma and Prafulla Dhariwal · 2018
Cited alongside, same era.
Mitigating unwanted biases with adversarial learning
Brian Hu Zhang, Blake Lemoine, and Margaret Mitchell · 2018
Cited alongside, same era.
Et-rnn: Applying deep learning to credit loan applications
Dmitrii Babaev, Maxim Savchenko, Alexander Tuzhilin, and Dmitrii Umerenkov · 2019
Cited alongside, same era.
Arcface: Additive angular margin loss for deep face recognition
Jiankang Deng, Jia Guo, Niannan Xue, and Stefanos Zafeiriou · 2019
Cited alongside, same era.
Fairface: Face attribute dataset for balanced race, gender, and age
Original
Kimmo Kärkkäinen and Jungseock Joo · 2019
Cited alongside, same era.

Then

Normalizing flows: An introduction and review of current methods
Ivan Kobyzev, Simon JD Prince, and Marcus A Brubaker · 2020
Later among the works it cites.
Interpreting the latent space of gans for semantic face editing
Yujun Shen, Jinjin Gu, Xiaoou Tang, and Bolei Zhou · 2020
Later among the works it cites.
Improving the fairness of deep generative models without retraining
Original
Shuhan Tan, Yujun Shen, and Bolei Zhou · 2020
Later among the works it cites.
Graph-based deep learning for medical diagnosis and analysis: past, present and future
David Ahmedt-Aristizabal, Mohammad Ali Armin, Simon Denman, Clinton Fookes, and Lars Petersson · 2021
Later among the works it cites.
Machine unlearning
Lucas Bourtoule, Varun Chandrasekaran, Christopher A Choquette-Choo, Hengrui Jia, Adelin Travers, Baiwu Zhang, David Lie, and Nicolas Papernot · 2021
Later among the works it cites.
Extracting training data from large language models
Nicholas Carlini, Florian Tramèr, Eric Wallace, Matthew Jagielski, Ariel Herbert-Voss, Katherine Lee, Adam Roberts, Tom Brown, Dawn Song, Úlfar Erlingsson, Alina Oprea, and Colin Raffel · 2021
Later among the works it cites.
When machine unlearning jeopardizes privacy
Min Chen, Zhikun Zhang, Tianhao Wang, Michael Backes, Mathias Humbert, and Yang Zhang · 2021
Later among the works it cites.
Navigating the gan parameter space for semantic image editing
Anton Cherepkov, Andrey Voynov, and Artem Babenko · 2021
Later among the works it cites.
https://www.washingtonpost.com/nation/2021/03/13/cheer-mom-deepfake-teammates/
The Washington Post · 2021
Later among the works it cites.
Fair attribute classification through latent space de-biasing
Vikram V Ramaswamy, Sunnie SY Kim, and Olga Russakovsky · 2021
Later among the works it cites.
Gan-control: Explicitly controllable gans
Alon Shoshan, Nadav Bhonker, Igor Kviatkovsky, and Gerard Medioni · 2021
Later among the works it cites.
Reconstructing training data from trained neural networks
Original
Niv Haim, Gal Vardi, Gilad Yehudai, Ohad Shamir, and Michal Irani · 2022
Closest in time.
Fairstyle: Debiasing stylegan2 with style channel manipulations
Cemre Efe Karakas, Alara Dirik, Eylül Yalçınkaya, and Pinar Yanardag · 2022
Closest in time.
Forgetting data from pre-trained gans
Original
Zhifeng Kong and Kamalika Chaudhuri · 2022
Closest in time.
Repairing neural networks by leaving the right past behind
Original
Ryutaro Tanno, Melanie F Pradier, Aditya Nori, and Yingzhen Li · 2022
Closest in time.
Rewriting geometric rules of a gan
Sheng-Yu Wang, David Bau, and Jun-Yan Zhu · 2022
Closest in time.
Generative visual prompt: Unifying distributional control of pre-trained generative models
Original
Chen Henry Wu, Saman Motamed, Shaunak Srivastava, and Fernando De la Torre · 2022
Closest in time.
Puma: Performance unchanged model augmentation for training data removal
Ga Wu, Masoud Hashemi, and Christopher Srinivasa · 2022
Closest in time.
Extracting training data from diffusion models
Original
Nicholas Carlini, Jamie Hayes, Milad Nasr, Matthew Jagielski, Vikash Sehwag, Florian Tramèr, Borja Balle, Daphne Ippolito, and Eric Wallace · 2023
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…