Byzantine Robustness and Partial Participation Can Be Achieved at Once: Just Clip Gradient Differences

Understand

Distributed learning has emerged as a leading paradigm for training large machine learning models.

• However, in real-world scenarios, participants may be unreliable or malicious, posing a significant challenge to the integrity and accuracy of the trained models.
• Byzantine fault tolerance mechanisms have been proposed to address these issues, but they often assume full participation from all clients, which is not always practical due to the unavailability of some clients or communication constraints.
• In our work, we propose the first distributed method with client sampling and provable tolerance to Byzantine workers.