Scaling up Differentially Private Deep Learning with Fast Per-Example Gradient Clipping

Understand

Recent work on Renyi Differential Privacy has shown the feasibility of applying differential privacy to deep learning tasks.

• Despite their promise, however, differentially private deep networks often lag far behind their non-private counterparts in accuracy, showing the need for more research in model architectures, optimizers, etc.
• One of the barriers to this expanded research is the training time -- often orders of magnitude larger than training non-private networks.
• The reason for this slowdown is a crucial privacy-related step called "per-example gradient clipping" whose naive implementation undoes the benefits of batch training with GPUs.