Hyperparameter Tuning with Renyi Differential Privacy

Understand

For many differentially private algorithms, such as the prominent noisy stochastic gradient descent (DP-SGD), the analysis needed to bound the privacy leakage of a single training run is well understood.

• However, few studies have reasoned about the privacy leakage resulting from the multiple training runs needed to fine tune the value of the training algorithm's hyperparameters.
• In this work, we first illustrate how simply setting hyperparameters based on non-private training runs can leak private information.
• Motivated by this observation, we then provide privacy guarantees for hyperparameter search procedures within the framework of Renyi Differential Privacy.