SPD-CFL: Stepwise Parameter Dropout for Efficient Continual Federated Learning

J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei, “Imagenet: A large-scale hierarchical image database,” in 2009 IEEE conference on computer vision and pattern recognition . Ieee, 2009, pp. 248–255

2009

Earlier work this paper cites.

J. Kirkpatrick, R. Pascanu, N. Rabinowitz, J. Veness, G. Desjardins, A. A. Rusu, K. Milan, J. Quan, T. Ramalho, A. Grabska-Barwinska et al. , “Overcoming catastrophic forgetting in neural networks,” Proceedings of the national academy of sciences , vol. 114, no. 13, pp. 3521–3526, 2017

2017

Earlier work this paper cites.

Z. Li and D. Hoiem, “Learning without forgetting,” IEEE transactions on pattern analysis and machine intelligence , vol. 40, no. 12, pp. 2935–2947, 2017

2017

Earlier work this paper cites.

B. McMahan, E. Moore, D. Ramage, S. Hampson, and B. A. y Arcas, “Communication-efficient learning of deep networks from decentralized data,” in Artificial intelligence and statistics . PMLR, 2017, pp. 1273–1282

2017

Earlier work this paper cites.

R. Aljundi, F. Babiloni, M. Elhoseiny, M. Rohrbach, and T. Tuytelaars, “Memory aware synapses: Learning what (not) to forget,” in Proceedings of the European conference on computer vision (ECCV) , 2018, pp. 139–154

2018

Earlier work this paper cites.

P. Kairouz, H. B. McMahan et al. , “Advances and open problems in federated learning,” Foundations and Trends in Machine Learning , vol. 14, no. 1-2, pp. 1–210, 2021

2021

Earlier work this paper cites.

S. Guo, T. Zhang, H. Yu, X. Xie, L. Ma, T. Xiang, and Y. Liu, “Byzantine-resilient decentralized stochastic gradient descent,” IEEE Transactions on Circuits and Systems for Video Technology , vol. 32, no. 6, pp. 4096–4106, 2021

2021

Earlier work this paper cites.

O. Shahid, S. Pouriyeh, R. M. Parizi, Q. Z. Sheng, G. Srivastava, and L. Zhao, “Communication efficiency in federated learning: Achievements and challenges,” arXiv preprint arXiv:2107.10996 , 2021

Original

2021

Earlier work this paper cites.

M. Gheini, X. Ren, and J. May, “Cross-attention is all you need: Adapting pretrained transformers for machine translation,” arXiv preprint arXiv:2104.08771 , 2021

Original

2021

Earlier work this paper cites.

L. Wang, M. Zhang, Z. Jia, Q. Li, C. Bao, K. Ma, J. Zhu, and Y. Zhong, “Afec: Active forgetting of negative transfer in continual learning,” Advances in Neural Information Processing Systems , vol. 34, pp. 22 379–22 391, 2021

2021

Earlier work this paper cites.

Z. Liu, Y. Lin, Y. Cao, H. Hu, Y. Wei, Z. Zhang, S. Lin, and B. Guo, “Swin transformer: Hierarchical vision transformer using shifted windows,” in Proceedings of the IEEE/CVF international conference on computer vision , 2021, pp. 10 012–10 022

2021

Earlier work this paper cites.

E. B. Zaken, S. Ravfogel, and Y. Goldberg, “Bitfit: Simple parameter-efficient fine-tuning for transformer-based masked language-models,” arXiv preprint arXiv:2106.10199 , 2021

Original

2021

Earlier work this paper cites.

L. Qu, Y. Zhou, P. P. Liang, Y. Xia, F. Wang, E. Adeli, L. Fei-Fei, and D. Rubin, “Rethinking architecture design for tackling data heterogeneity in federated learning,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2022, pp. 10 061–10 071

2022

Earlier work this paper cites.

H.-Y. Chen, C.-H. Tu, Z. Li, H.-W. Shen, and W.-L. Chao, “On the importance and applicability of pre-training for federated learning,” arXiv preprint arXiv:2206.11488 , 2022

Original

2022

Earlier work this paper cites.

Y. Tian, Y. Wan, L. Lyu, D. Yao, H. Jin, and L. Sun, “Fedbert: When federated learning meets pre-training,” ACM Transactions on Intelligent Systems and Technology (TIST) , vol. 13, no. 4, pp. 1–26, 2022

2022

Earlier work this paper cites.

G. Sun, M. Mendieta, T. Yang, and C. Chen, “Conquering the communication constraints to enable large pre-trained models in federated learning,” arXiv , 2022

2022

Earlier work this paper cites.

J. Chen, W. Xu, S. Guo, J. Wang, J. Zhang, and H. Wang, “Fedtune: A deep dive into efficient federated fine-tuning with pre-trained transformers,” arXiv preprint arXiv:2211.08025 , 2022

Original

2022

Earlier work this paper cites.

E. J. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, W. Chen et al. , “Lora: Low-rank adaptation of large language models.” ICLR , vol. 1, no. 2, p. 3, 2022

2022

Earlier work this paper cites.

D. Cai, Y. Wu, S. Wang, F. X. Lin, and M. Xu, “Fedadapter: Efficient federated learning for modern nlp,” arXiv preprint arXiv:2205.10162 , 2022

Original

2022

Cited alongside, same era.

R. Tiwari, K. Killamsetty, R. Iyer, and P. Shenoy, “Gcr: Gradient coreset based replay buffer selection for continual learning,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2022, pp. 99–108

2022

Cited alongside, same era.

Y. Zhang, B. Pfahringer, E. Frank, A. Bifet, N. J. S. Lim, and Y. Jia, “A simple but strong baseline for online continual learning: Repeated augmented rehearsal,” Advances in Neural Information Processing Systems , vol. 35, pp. 14 771–14 783, 2022

2022

Cited alongside, same era.

Q. Zhang, S. Zuo, C. Liang, A. Bukharin, P. He, W. Chen, and T. Zhao, “Platon: Pruning large transformer models with upper confidence bound of weight importance,” in International conference on machine learning . PMLR, 2022, pp. 26 809–26 823

2022

Q. Gao, X. Shan, Y. Zhang, and F. Zhou, “Enhancing knowledge transfer for task incremental learning with data-free subnetwork,” Advances in Neural Information Processing Systems , vol. 36, pp. 68 471–68 484, 2023

2023

Later among the works it cites.

C. Lamers, R. Vidal, N. Belbachir, N. van Stein, T. Bäeck, and P. Giampouras, “Clustering-based domain-incremental learning,” in Proceedings of the IEEE/CVF International Conference on Computer Vision , 2023, pp. 3384–3392

2023

Later among the works it cites.

B. Wickramasinghe, G. Saha, and K. Roy, “Continual learning: A review of techniques, challenges and future directions,” IEEE Transactions on Artificial Intelligence , vol. 5, no. 6, pp. 2526–2546, 2023

2023

Later among the works it cites.

C.-H. Kao and Y.-C. F. Wang, “Fedbug: A bottom-up gradual unfreezing framework for federated learning,” arXiv preprint arXiv:2307.10317 , 2023

Original

2023

Later among the works it cites.