Differentially Private Knowledge Distillation via Synthetic Text Generation

Understand

Large Language models (LLMs) are achieving state-of-the-art performance in many different downstream tasks.

• However, the increasing urgency of data privacy puts pressure on practitioners to train LLMs with Differential Privacy (DP) on private data.
• Concurrently, the exponential growth in parameter size of LLMs necessitates model compression before deployment of LLMs on resource-constrained devices or latency-sensitive applications.
• Differential privacy and model compression generally must trade off utility loss to achieve their objectives.