Exploring Parameter-Efficient Fine-Tuning to Enable Foundation Models in Federated Learning

Understand

Federated learning (FL) has emerged as a promising paradigm for enabling the collaborative training of models without centralized access to the raw data on local devices.

• In the typical FL paradigm (e.g., FedAvg), model weights are sent to and from the server each round to participating clients.
• Recently, the use of small pre-trained models has been shown to be effective in federated learning optimization and improving convergence.
• However, recent state-of-the-art pre-trained models are getting more capable but also have more parameters, known as the "Foundation Models." In conventional FL, sharing the enormous model weights can quickly put a massive communication burden on the system, especially if more capable models are employed.