2022

FedPrompt: Communication-Efficient and Privacy Preserving Prompt Tuning in Federated Learning

Zhao, Haodong, Du, Wei, Li, Fangqi et al.

Understand

Federated learning (FL) has enabled global model training on decentralized data in a privacy-preserving way by aggregating model updates.

However, for many natural language processing (NLP) tasks that utilize pre-trained language models (PLMs) with large numbers of parameters, there are considerable communication costs associated with FL.
Recently, prompt tuning, which tunes some soft prompts without modifying PLMs, has achieved excellent performance as a new learning paradigm.
Therefore we want to combine the two methods and explore the effect of prompt tuning under FL.

Built on

G. Sakkis, I. Androutsopoulos, G. Paliouras, V. Karkaletsis, C. D. Spyropoulos, and P. Stamatopoulos, “A memory-based approach to anti-spam filtering for mailing lists,” Inf. Retr. , vol. 6, no. 1, pp. 49–73, 2003
2003
Earlier work this paper cites.
V. Metsis, I. Androutsopoulos, and G. Paliouras, “Spam filtering with naive bayes - which naive bayes?” in CEAS , 2006
2006
Earlier work this paper cites.
J. Dean, G. Corrado, R. Monga, K. Chen, M. Devin, Q. V. Le, M. Z. Mao, M. Ranzato, A. W. Senior, P. A. Tucker, K. Yang, and A. Y. Ng, “Large scale distributed deep networks,” in Advances in Neural Information Processing Systems , 2012, pp. 1232–1240
2012
Earlier work this paper cites.
M. Li, L. Zhou, Z. Yang, A. Q. Li, F. Xia, D. G. Andersen, and A. Smola, “Parameter server for distributed machine learning,” 2013
2013
Earlier work this paper cites.
K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition . IEEE Computer Society, 2016, pp. 770–778
2016
Earlier work this paper cites.
M. Abadi, A. Chu, I. J. Goodfellow, H. B. McMahan, I. Mironov, K. Talwar, and L. Zhang, “Deep learning with differential privacy,” in Proceedings of the 2016 ACM SIGSAC Conference on Computer and Communications Securit , 2016, pp. 308–318
2016
Earlier work this paper cites.
P. Rajpurkar, J. Zhang, K. Lopyrev, and P. Liang, “Squad: 100, 000+ questions for machine comprehension of text,” in Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing , 2016, pp. 2383–2392
2016
Earlier work this paper cites.
B. McMahan, E. Moore, D. Ramage, S. Hampson, and B. A. y Arcas, “Communication-efficient learning of deep networks from decentralized data,” in Artificial Intelligence and Statistics . PMLR, 2017, pp. 1273–1282
2017
Earlier work this paper cites.
R. C. Geyer, T. Klein, and M. Nabi, “Differentially Private Federated Learning: A Client Level Perspective,” p. arXiv:1712.07557, Dec. 2017
Original
2017
Earlier work this paper cites.
K. A. Bonawitz, V. Ivanov, B. Kreuter, A. Marcedone, H. B. McMahan, S. Patel, D. Ramage, A. Segal, and K. Seth, “Practical secure aggregation for privacy-preserving machine learning,” in Proceedings of the ACM SIGSAC Conference on Computer and Communications Security , 2017, pp. 1175–1191
2017
Earlier work this paper cites.
H. B. McMahan, D. Ramage, K. Talwar, and L. Zhang, “Learning differentially private recurrent language models,” in International Conference on Learning Representations , 2018
2018
Earlier work this paper cites.
A. Founta, C. Djouvas, D. Chatzakou, I. Leontiadis, J. Blackburn, G. Stringhini, A. Vakali, M. Sirivianos, and N. Kourtellis, “Large scale crowdsourcing and characterization of twitter abusive behavior,” in Proceedings of the Twelfth International Conference on Web and Social Media , 2018, pp. 491–500
2018
Earlier work this paper cites.
J. Devlin, M. Chang, K. Lee, and K. Toutanova, “BERT: pre-training of deep bidirectional transformers for language understanding,” in NAACL-HLT , 2019, pp. 4171–4186
2019
Earlier work this paper cites.
Y. Liu, M. Ott, N. Goyal, J. Du, M. Joshi, D. Chen, O. Levy, M. Lewis, L. Zettlemoyer, and V. Stoyanov, “Roberta: A robustly optimized BERT pretraining approach,” p. arXiv:1907.11692, 2019
Original
2019
Earlier work this paper cites.
Z. Wu, Y. Zhou, D. Wu, M. Chen, and Y. Xu, “TAMF: towards personalized time-aware recommendation for over-the-top videos,” in Proceedings of the ACM Workshop on Network and Operating Systems Support for Digital Audio and Video , 2019, pp. 43–48
2019
Earlier work this paper cites.
Q. Yang, Y. Liu, T. Chen, and Y. Tong, “Federated machine learning: Concept and applications,” ACM Trans. Intell. Syst. Technol. , vol. 10, no. 2, pp. 12:1–12:19, 2019
2019
Earlier work this paper cites.

Similar

L. Melis, C. Song, E. D. Cristofaro, and V. Shmatikov, “Exploiting unintended feature leakage in collaborative learning,” in IEEE Symposium on Security and Privacy . IEEE, 2019, pp. 691–706
2019
Cited alongside, same era.
A. Triastcyn and B. Faltings, “Federated learning with bayesian differential privacy,” in IEEE International Conference on Big Data (IEEE BigData) . IEEE, 2019, pp. 2587–2596
2019
Cited alongside, same era.
X. Yuan, P. He, Q. Zhu, and X. Li, “Adversarial examples: Attacks and defenses for deep learning,” IEEE Trans. Neural Networks Learn. Syst. , vol. 30, no. 9, pp. 2805–2824, 2019
2019
Cited alongside, same era.
M. Zampieri, S. Malmasi, P. Nakov, S. Rosenthal, N. Farra, and R. Kumar, “Semeval-2019 task 6: Identifying and categorizing offensive language in social media (offenseval),” in Proceedings of the 13th International Workshop on Semantic Evaluation, SemEval@NAACL-HLT 2019 , 2019, pp. 75–86
2019
Cited alongside, same era.
C. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y. Zhou, W. Li, and P. J. Liu, “Exploring the limits of transfer learning with a unified text-to-text transformer,” J. Mach. Learn. Res. , vol. 21, pp. 140:1–140:67, 2020
2020
Cited alongside, same era.
J. Hamer, M. Mohri, and A. T. Suresh, “Fedboost: A communication-efficient algorithm for federated learning,” in Proceedings of the International Conference on Machine Learning . PMLR, 2020, pp. 3973–3983
2020
Cited alongside, same era.
T. Qi, F. Wu, C. Wu, Y. Huang, and X. Xie, “Privacy-preserving news recommendation model learning,” in Findings of the Association for Computational Linguistics: EMNLP , 2020, pp. 1423–1432
2020
Cited alongside, same era.
T. Li, A. K. Sahu, M. Zaheer, M. Sanjabi, A. Talwalkar, and V. Smith, “Federated optimization in heterogeneous networks,” in Proceedings of Machine Learning and Systems , 2020
2020
Cited alongside, same era.
S. P. Karimireddy, S. Kale, M. Mohri, S. J. Reddi, S. U. Stich, and A. T. Suresh, “SCAFFOLD: stochastic controlled averaging for federated learning,” in Proceedings of the International Conference on Machine Learning . PMLR, 2020, pp. 5132–5143
2020
Cited alongside, same era.
A. Fallah, A. Mokhtari, and A. E. Ozdaglar, “Personalized federated learning with theoretical guarantees: A model-agnostic meta-learning approach,” in Advances in Neural Information Processing System , 2020
2020
Cited alongside, same era.
J. Geiping, H. Bauermeister, H. Dröge, and M. Moeller, “Inverting gradients - how easy is it to break privacy in federated learning?” in Advances in Neural Information Processing System , 2020
2020
Cited alongside, same era.
E. Bagdasaryan, A. Veit, Y. Hua, D. Estrin, and V. Shmatikov, “How to backdoor federated learning,” in Artificial Intelligence and Statistics . PMLR, 2020, pp. 2938–2948
2020
Cited alongside, same era.

Then

X. Liu, Y. Zheng, Z. Du, M. Ding, Y. Qian, Z. Yang, and J. Tang, “Gpt understands, too,” p. arXiv:2103.10385, 2021
Original
2021
Later among the works it cites.
X. Liu, K. Ji, Y. Fu, Z. Du, Z. Yang, and J. Tang, “P-tuning v2: Prompt tuning can be comparable to fine-tuning universally across scales and tasks,” p. arXiv:2110.07602, 2021
Original
2021
Later among the works it cites.
X. Han, W. Zhao, N. Ding, Z. Liu, and M. Sun, “Ptr: Prompt tuning with rules for text classification,” p. arXiv:2105.11259, 2021
Original
2021
Later among the works it cites.
D. A. E. Acar, Y. Zhao, R. M. Navarro, M. Mattina, P. N. Whatmough, and V. Saligrama, “Federated learning based on dynamic regularization,” in International Conference on Learning Representations , 2021
2021
Later among the works it cites.
X. Li and D. Zhan, “Fedrs: Federated learning with restricted softmax for label distribution non-iid data,” in KDD ’21: The ACM SIGKDD Conference on Knowledge Discovery and Data Mining , 2021, pp. 995–1005
2021
Later among the works it cites.
J. Yi, F. Wu, C. Wu, R. Liu, G. Sun, and X. Xie, “Efficient-fedrec: Efficient federated learning framework for privacy-preserving news recommendation,” in Proceedings of the Conference on Empirical Methods in Natural Language Processing , 2021, pp. 2814–2824
2021
Later among the works it cites.
M. Jere, T. Farnan, and F. Koushanfar, “A taxonomy of attacks on federated learning,” IEEE Secur. Priv. , vol. 19, no. 2, pp. 20–28, 2021
2021
Later among the works it cites.
Q. Li, B. He, and D. Song, “Model-contrastive federated learning,” in IEEE Conference on Computer Vision and Pattern Recognition , 2021, pp. 10 713–10 722
2021
Later among the works it cites.
A. Hilmkil, S. Callh, M. Barbieri, L. R. Sütfeld, E. L. Zec, and O. Mogren, “Scaling federated learning for fine-tuning of large language models,” in International Conference on Applications of Natural Language to Information Systems , 2021, pp. 15–23
2021
Later among the works it cites.
P. Basu, T. S. Roy, R. Naidu, Z. Müftüoglu, S. Singh, and F. Mireshghallah, “Benchmarking differential privacy and federated learning for BERT models,” p. arXiv:2106.13973, 2021
Original
2021
Later among the works it cites.
Y. Gu, X. Han, Z. Liu, and M. Huang, “PPT: pre-trained prompt tuning for few-shot learning,” in Proceedings of the Annual Meeting of the Association for Computational Linguistics , 2022, pp. 8410–8423
2022
Closest in time.
A. Aldahdooh, W. Hamidouche, S. A. Fezza, and O. Déforges, “Adversarial example detection for DNN models: a review and experimental comparison,” Artif. Intell. Rev. , vol. 55, no. 6, pp. 4403–4462, 2022
2022
Closest in time.
W. Du, Y. Zhao, B. Li, G. Liu, and S. Wang, “Ppt: Backdoor attacks on pre-trained models via poisoned prompt tuning,” in IJCAI-22 , 2022, pp. 680–686
2022
Closest in time.
C. He, E. Ceyani, K. Balasubramanian, M. Annavaram, and S. Avestimehr, “Spreadgnn: Decentralized multi-task federated learning for graph neural networks on molecular data,” in Proceedings of the AAAI Conference on Artificial Intelligence , 2022, pp. 6865–6873
2022
Closest in time.
A. Reisizadeh, A. Mokhtari, H. Hassani, A. Jadbabaie, and R. Pedarsani, “Fedpaq: A communication-efficient federated learning method with periodic averaging and quantization,” in Artificial Intelligence and Statistics . PMLR, 2020, pp. 2021–2031
2031
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…