Built on

Bayesian variable selection in linear regression
Toby J Mitchell and John J Beauchamp · 1988
Earlier work this paper cites.
Equivalence of regularization and truncated iteration for general ill-posed problems
Reginaldo J Santos · 1996
Earlier work this paper cites.
Gradient-based learning applied to document recognition
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner · 1998
Earlier work this paper cites.
A view of the em algorithm that justifies incremental, sparse, and other variants
Radford M Neal and Geoffrey E Hinton · 1998
Earlier work this paper cites.
Adam: A method for stochastic optimization
Original
Diederik P Kingma and Jimmy Ba · 2014
Earlier work this paper cites.
Communication-efficient learning of deep networks from decentralized data
Original
H Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, et al · 2016
Earlier work this paper cites.
Learning structured sparsity in deep neural networks
Original
Wei Wen, Chunpeng Wu, Yandan Wang, Yiran Chen, and Hai Li · 2016
Earlier work this paper cites.
Deep gradient compression: Reducing the communication bandwidth for distributed training
Original
Yujun Lin, Song Han, Huizi Mao, Yu Wang, and William J Dally · 2017
Earlier work this paper cites.
Learning sparse neural networks through l _ 0 l\_0 regularization
Original
Christos Louizos, Max Welling, and Diederik P Kingma · 2017
Earlier work this paper cites.
signsgd: Compressed optimisation for non-convex problems
Original
Jeremy Bernstein, Yu-Xiang Wang, Kamyar Azizzadenesheli, and Anima Anandkumar · 2018
Earlier work this paper cites.
Expanding the reach of federated learning by reducing client resource requirements
Original
Sebastian Caldas, Jakub Konečny, H Brendan McMahan, and Ameet Talwalkar · 2018
Earlier work this paper cites.
Recasting gradient-based meta-learning as hierarchical bayes
Original
Erin Grant, Chelsea Finn, Sergey Levine, Trevor Darrell, and Thomas Griffiths · 2018
Earlier work this paper cites.

Similar

Federated optimization in heterogeneous networks
Original
Tian Li, Anit Kumar Sahu, Manzil Zaheer, Maziar Sanjabi, Ameet Talwalkar, and Virginia Smith · 2018
Cited alongside, same era.
On first-order meta-learning algorithms
Original
Alex Nichol, Joshua Achiam, and John Schulman · 2018
Cited alongside, same era.
Gradient sparsification for communication-efficient distributed optimization
Jianqiao Wangni, Jialei Wang, Ji Liu, and Tong Zhang · 2018
Cited alongside, same era.
Modular meta-learning with shrinkage
Original
Yutian Chen, Abram L Friesen, Feryal Behbahani, David Budden, Matthew W Hoffman, Arnaud Doucet, and Nando de Freitas · 2019
Cited alongside, same era.
Variational federated multi-task learning
Original

Then

Bayesian nonparametric federated learning of neural networks
Mikhail Yurochkin, Mayank Agarwal, Soumya Ghosh, Kristjan Greenewald, Nghia Hoang, and Yasaman Khazaeni · 2019
Later among the works it cites.
Statistical model aggregation via parameter matching
Original
Mikhail Yurochkin, Mayank Agarwal, Soumya Ghosh, Kristjan Greenewald, and Trong Nghia Hoang · 2019
Later among the works it cites.
Lookahead optimizer: k steps forward, 1 step back
Original
Michael Zhang, James Lucas, Jimmy Ba, and Geoffrey E Hinton · 2019
Later among the works it cites.
Federated learning via posterior averaging: A new perspective and practical algorithms
Original
Maruan Al-Shedivat, Jennifer Gillenwater, Eric Xing, and Afshin Rostamizadeh · 2020
Later among the works it cites.
Federated learning with quantized global model updates

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…

2021

An Expectation-Maximization Perspective on Federated Learning

Louizos, Christos, Reisser, Matthias, Soriaga, Joseph et al.

Understand

Federated learning describes the distributed training of models across multiple clients while keeping the data private on-device.

In this work, we view the server-orchestrated federated learning process as a hierarchical latent variable model where the server provides the parameters of a prior distribution over the client-specific model parameters.
We show that with simple Gaussian priors and a hard version of the well known Expectation-Maximization (EM) algorithm, learning in such a model corresponds to FedAvg, the most popular algorithm for the federated learning setting.
This perspective on FedAvg unifies several recent works in the field and opens up the possibility for extensions through different choices for the hierarchical model.

Built on

Bayesian variable selection in linear regression
Toby J Mitchell and John J Beauchamp · 1988
Earlier work this paper cites.
Equivalence of regularization and truncated iteration for general ill-posed problems
Reginaldo J Santos · 1996
Earlier work this paper cites.
Gradient-based learning applied to document recognition
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner · 1998
Earlier work this paper cites.
A view of the em algorithm that justifies incremental, sparse, and other variants
Radford M Neal and Geoffrey E Hinton · 1998
Earlier work this paper cites.
Adam: A method for stochastic optimization
Original
Diederik P Kingma and Jimmy Ba · 2014
Earlier work this paper cites.
Communication-efficient learning of deep networks from decentralized data
Original
H Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, et al · 2016
Earlier work this paper cites.
Learning structured sparsity in deep neural networks
Original
Wei Wen, Chunpeng Wu, Yandan Wang, Yiran Chen, and Hai Li · 2016
Earlier work this paper cites.
Deep gradient compression: Reducing the communication bandwidth for distributed training
Original
Yujun Lin, Song Han, Huizi Mao, Yu Wang, and William J Dally · 2017
Earlier work this paper cites.
Learning sparse neural networks through l _ 0 l\_0 regularization
Original
Christos Louizos, Max Welling, and Diederik P Kingma · 2017
Earlier work this paper cites.
signsgd: Compressed optimisation for non-convex problems
Original
Jeremy Bernstein, Yu-Xiang Wang, Kamyar Azizzadenesheli, and Anima Anandkumar · 2018
Earlier work this paper cites.
Expanding the reach of federated learning by reducing client resource requirements
Original
Sebastian Caldas, Jakub Konečny, H Brendan McMahan, and Ameet Talwalkar · 2018
Earlier work this paper cites.
Recasting gradient-based meta-learning as hierarchical bayes
Original
Erin Grant, Chelsea Finn, Sergey Levine, Trevor Darrell, and Thomas Griffiths · 2018
Earlier work this paper cites.

Similar

Federated optimization in heterogeneous networks
Original
Tian Li, Anit Kumar Sahu, Manzil Zaheer, Maziar Sanjabi, Ameet Talwalkar, and Virginia Smith · 2018
Cited alongside, same era.
On first-order meta-learning algorithms
Original
Alex Nichol, Joshua Achiam, and John Schulman · 2018
Cited alongside, same era.
Gradient sparsification for communication-efficient distributed optimization
Jianqiao Wangni, Jialei Wang, Ji Liu, and Tong Zhang · 2018
Cited alongside, same era.
Modular meta-learning with shrinkage
Original
Yutian Chen, Abram L Friesen, Feryal Behbahani, David Budden, Matthew W Hoffman, Arnaud Doucet, and Nando de Freitas · 2019
Cited alongside, same era.
Variational federated multi-task learning
Original

Then

Bayesian nonparametric federated learning of neural networks
Mikhail Yurochkin, Mayank Agarwal, Soumya Ghosh, Kristjan Greenewald, Nghia Hoang, and Yasaman Khazaeni · 2019
Later among the works it cites.
Statistical model aggregation via parameter matching
Original
Mikhail Yurochkin, Mayank Agarwal, Soumya Ghosh, Kristjan Greenewald, and Trong Nghia Hoang · 2019
Later among the works it cites.
Lookahead optimizer: k steps forward, 1 step back
Original
Michael Zhang, James Lucas, Jimmy Ba, and Geoffrey E Hinton · 2019
Later among the works it cites.
Federated learning via posterior averaging: A new perspective and practical algorithms
Original
Maruan Al-Shedivat, Jennifer Gillenwater, Eric Xing, and Afshin Rostamizadeh · 2020
Later among the works it cites.
Federated learning with quantized global model updates

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Luca Corinzia, Ami Beuret, and Joachim M Buhmann · 2019

Cited alongside, same era.

The state of sparsity in deep neural networks

Original

Trevor Gale, Erich Elsen, and Sara Hooker · 2019

Cited alongside, same era.

Measuring the effects of non-identical data distribution for federated visual classification

Original

Tzu-Ming Harry Hsu, Hang Qi, and Matthew Brown · 2019

Cited alongside, same era.

Improving federated learning personalization via model agnostic meta learning

Original

Yihan Jiang, Jakub Konečnỳ, Keith Rush, and Sreeram Kannan · 2019

Cited alongside, same era.

Advances and open problems in federated learning

Original

Peter Kairouz, H Brendan McMahan, Brendan Avent, Aurélien Bellet, Mehdi Bennis, Arjun Nitin Bhagoji, Keith Bonawitz, Zachary Charles, Graham Cormode, Rachel Cummings, et al · 2019

Cited alongside, same era.

Pytorch: An imperative style, high-performance deep learning library

Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala · 2019

Cited alongside, same era.

Robust and communication-efficient federated learning from non-iid data

Felix Sattler, Simon Wiedemann, Klaus-Robert Müller, and Wojciech Samek · 2019

Cited alongside, same era.

Double quantization for communication-efficient distributed optimization

Yue Yu, Jiaxiang Wu, and Longbo Huang · 2019

Cited alongside, same era.

Original

Mohammad Mohammadi Amiri, Deniz Gunduz, Sanjeev R Kulkarni, and H Vincent Poor · 2020