Fetching the paper…

Reading the bibliography…

Achieving Personalized Federated Learning with Sparse Local Models · Around

2022

Achieving Personalized Federated Learning with Sparse Local Models

Huang, Tiansheng, Liu, Shiwei, Shen, Li et al.

Original
PDF
Post on X

Understand

Federated learning (FL) is vulnerable to heterogeneously distributed data, since a common global model in FL may not adapt to the heterogeneous data distribution of each user.

To counter this issue, personalized FL (PFL) was proposed to produce dedicated local models for each individual user.
However, PFL is far from its maturity, because existing PFL solutions either demonstrate unsatisfactory generalization towards different model architectures or cost enormous extra computation and memory.
In this work, we propose federated learning with personalized sparse mask (FedSpa), a novel PFL scheme that employs personalized sparse masks to customize sparse local models on the edge.

Built on

K. Simonyan and A. Zisserman, “Very deep convolutional networks for large-scale image recognition,” arXiv preprint arXiv:1409.1556 , 2014
Original
2014
Earlier work this paper cites.
Y. LeCun et al. , “Lenet-5, convolutional neural networks,” URL: http://yann. lecun. com/exdb/lenet , vol. 20, no. 5, p. 14, 2015
2015
Earlier work this paper cites.
S. Ioffe and C. Szegedy, “Batch normalization: Accelerating deep network training by reducing internal covariate shift,” in International conference on machine learning . PMLR, 2015, pp. 448–456
2015
Earlier work this paper cites.
H. B. McMahan, E. Moore, D. Ramage, S. Hampson et al. , “Communication-efficient learning of deep networks from decentralized data,” arXiv preprint arXiv:1602.05629 , 2016
Original
2016
Earlier work this paper cites.
K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in Proceedings of the IEEE conference on computer vision and pattern recognition , 2016, pp. 770–778
2016
Earlier work this paper cites.
J. Konečnỳ, H. B. McMahan, F. X. Yu, P. Richtárik, A. T. Suresh, and D. Bacon, “Federated learning: Strategies for improving communication efficiency,” arXiv preprint arXiv:1610.05492 , 2016
Original
2016
Earlier work this paper cites.
P. Voigt and A. Von dem Bussche, “The eu general data protection regulation (gdpr),” A Practical Guide, 1st Ed., Cham: Springer International Publishing , vol. 10, p. 3152676, 2017
2017
Earlier work this paper cites.
J. Frankle and M. Carbin, “The lottery ticket hypothesis: Finding sparse, trainable neural networks,” arXiv preprint arXiv:1803.03635 , 2018
Original
2018
Earlier work this paper cites.
G. Bellec, D. Kappel, W. Maass, and R. Legenstein, “Deep rewiring: Training very sparse deep networks,” in International Conference on Learning Representations , 2018
2018
Earlier work this paper cites.
D. C. Mocanu, E. Mocanu, P. Stone, P. H. Nguyen, M. Gibescu, and A. Liotta, “Scalable training of artificial neural networks with adaptive sparse connectivity inspired by network science,” Nature Communications , vol. 9, no. 1, p. 2383, 2018
2018
Earlier work this paper cites.
S. U. Stich, “Local sgd converges fast and communicates little,” arXiv preprint arXiv:1805.09767 , 2018
Original
2018
Earlier work this paper cites.
Y. Wu and K. He, “Group normalization,” in Proceedings of the European conference on computer vision (ECCV) , 2018, pp. 3–19
2018
Earlier work this paper cites.
M. G. Arivazhagan, V. Aggarwal, A. K. Singh, and S. Choudhary, “Federated learning with personalization layers,” arXiv preprint arXiv:1912.00818 , 2019
Original
2019
Earlier work this paper cites.

Similar

D. Li and J. Wang, “Fedmd: Heterogenous federated learning via model distillation,” 2019
2019
Cited alongside, same era.
H. Mostafa and X. Wang, “Parameter efficient training of deep convolutional neural networks by dynamic sparse reparameterization,” International Conference on Machine Learning , 2019
2019
Cited alongside, same era.
T. Dettmers and L. Zettlemoyer, “Sparse networks from scratch: Faster training without losing performance,” arXiv preprint arXiv:1907.04840 , 2019
Original
2019
Cited alongside, same era.
T.-M. H. Hsu, H. Qi, and M. Brown, “Measuring the effects of non-identical data distribution for federated visual classification,” arXiv preprint arXiv:1909.06335 , 2019
Original
2019
Cited alongside, same era.
A. Li, J. Sun, B. Wang, L. Duan, S. Li, Y. Chen, and H. Li, “Lotteryfl: Personalized and communication-efficient federated learning with lottery ticket hypothesis on non-iid datasets,” arXiv preprint arXiv:2008.03371 , 2020

Then

S. Liu, D. C. Mocanu, A. R. R. Matavalam, Y. Pei, and M. Pechenizkiy, “Sparse evolutionary deep learning with over one million artificial neurons on commodity hardware,” Neural Computing and Applications , pp. 1–16, 2020
2020
Later among the works it cites.
S. Jayakumar, R. Pascanu, J. Rae, S. Osindero, and E. Elsen, “Top-kast: Top-k always sparse training,” Advances in Neural Information Processing Systems , vol. 33, 2020
2020
Later among the works it cites.
C. He, S. Li, J. So, X. Zeng, M. Zhang, H. Wang, X. Wang, P. Vepakomma, A. Singh, H. Qiu et al. , “Fedml: A research library and benchmark for federated machine learning,” arXiv preprint arXiv:2007.13518 , 2020
Original
2020
Later among the works it cites.
K. Hsieh, A. Phanishayee, O. Mutlu, and P. Gibbons, “The non-iid data quagmire of decentralized machine learning,” in International Conference on Machine Learning . PMLR, 2020, pp. 4387–4398
2020
Later among the works it cites.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

2020

Cited alongside, same era.

P. P. Liang, T. Liu, L. Ziyin, N. B. Allen, R. P. Auerbach, D. Brent, R. Salakhutdinov, and L.-P. Morency, “Think locally, act globally: Federated learning with local and global representations,” arXiv preprint arXiv:2001.01523 , 2020

2020

Cited alongside, same era.

F. Hanzely and P. Richtárik, “Federated learning of a mixture of global and local models,” arXiv preprint arXiv:2002.05516 , 2020

2020

Cited alongside, same era.

Y. Mansour, M. Mohri, J. Ro, and A. T. Suresh, “Three approaches for personalization with applications to federated learning,” arXiv preprint arXiv:2002.10619 , 2020

2020

Cited alongside, same era.

Y. Deng, M. M. Kamani, and M. Mahdavi, “Adaptive personalized federated learning,” 2020

2020

Cited alongside, same era.

H. Yang, H. He, W. Zhang, and X. Cao, “Fedsteg: A federated transfer learning framework for secure image steganalysis,” IEEE Transactions on Network Science and Engineering , 2020

2020

Cited alongside, same era.

Y. Chen, X. Qin, J. Wang, C. Yu, and W. Gao, “Fedhealth: A federated transfer learning framework for wearable healthcare,” IEEE Intelligent Systems , vol. 35, no. 4, pp. 83–93, 2020

2020

Cited alongside, same era.

U. Evci, T. Gale, J. Menick, P. S. Castro, and E. Elsen, “Rigging the lottery: Making all tickets winners,” in International Conference on Machine Learning . PMLR, 2020, pp. 2943–2952

2020

Cited alongside, same era.

S. P. Karimireddy, S. Kale, M. Mohri, S. Reddi, S. Stich, and A. T. Suresh, “Scaffold: Stochastic controlled averaging for federated learning,” in International Conference on Machine Learning . PMLR, 2020, pp. 5132–5143

2020

Later among the works it cites.

S. Reddi, Z. Charles, M. Zaheer, Z. Garrett, K. Rush, J. Konečnỳ, S. Kumar, and H. B. McMahan, “Adaptive federated optimization,” arXiv preprint arXiv:2003.00295 , 2020

2020

Later among the works it cites.

T. Li, S. Hu, A. Beirami, and V. Smith, “Ditto: Fair and robust federated learning through personalization,” 2021

2021

Later among the works it cites.

S. Vahidian, M. Morafah, and B. Lin, “Personalized federated learning by structured and unstructured pruning under data heterogeneity,” arXiv preprint arXiv:2105.00562 , 2021

2021

Later among the works it cites.

L. Collins, H. Hassani, A. Mokhtari, and S. Shakkottai, “Exploiting shared representations for personalized federated learning,” arXiv preprint arXiv:2102.07078 , 2021

Original

2021

Later among the works it cites.

S. Liu, L. Yin, D. C. Mocanu, and M. Pechenizkiy, “Do we actually need dense over-parameterization? in-time over-parameterization in sparse training,” in Proceedings of the 39th International Conference on Machine Learning . PMLR, 2021, pp. 6989–7000

2021

Later among the works it cites.

J. Xu, S. Wang, L. Wang, and A. C.-C. Yao, “Fedcm: Federated learning with client-level momentum,” arXiv preprint arXiv:2106.10874 , 2021

2021

Later among the works it cites.

A. Mohtashami, M. Jaggi, and S. U. Stich, “Simultaneous training of partially masked neural networks,” arXiv preprint arXiv:2106.08895 , 2021

2021

Later among the works it cites.