Enabling Differentially Private Federated Learning for Speech Recognition: Benchmarks, Adaptive Optimizers and Gradient Clipping

Understand

While federated learning (FL) and differential privacy (DP) have been extensively studied, their application to automatic speech recognition (ASR) remains largely unexplored due to the challenges in training large transformer models.

• Specifically, large models further exacerbate issues in FL as they are particularly susceptible to gradient heterogeneity across layers, unlike the relatively uniform gradient behavior observed in shallow models.
• As a result, prior works struggle to converge with standard optimization techniques, even in the absence of DP mechanisms.
• To the best of our knowledge, no existing work establishes a competitive, practical recipe for FL with DP in the context of ASR.