Differential Privacy Meets Neural Network Pruning

Understand

A major challenge in applying differential privacy to training deep neural network models is scalability.The widely-used training algorithm, differentially private stochastic gradient descent (DP-SGD), struggles with training moderately-sized neural network models for a value of epsilon corresponding to a high level of privacy protection.

• In this paper, we explore the idea of dimensionality reduction inspired by neural network pruning to improve the scalability of DP-SGD.
• We study the interplay between neural network pruning and differential privacy, through the two modes of parameter updates.
• We call the first mode, parameter freezing, where we pre-prune the network and only update the remaining parameters using DP-SGD.