Privacy-preserving Fine-tuning of Large Language Models through Flatness

Understand

The privacy concerns associated with the use of Large Language Models (LLMs) have grown recently with the development of LLMs such as ChatGPT.

• Differential Privacy (DP) techniques are explored in existing work to mitigate their privacy risks at the cost of generalization degradation.
• Our paper reveals that the flatness of DP-trained models' loss landscape plays an essential role in the trade-off between their privacy and generalization.
• We further propose a holistic framework to enforce appropriate weight flatness, which substantially improves model generalization with competitive privacy preservation.