Privately Customizing Prefinetuning to Better Match User Data in Federated Learning

Understand

In Federated Learning (FL), accessing private client data incurs communication and privacy costs.

• As a result, FL deployments commonly prefinetune pretrained foundation models on a (large, possibly public) dataset that is held by the central server; they then FL-finetune the model on a private, federated dataset held by clients.
• Evaluating prefinetuning dataset quality reliably and privately is therefore of high importance.
• To this end, we propose FreD (Federated Private Fr\'echet Distance) -- a privately computed distance between a prefinetuning dataset and federated datasets.