Fetching the paper…
Reading the bibliography…
We study (differentially) private federated learning (FL) of language models.
Language models are few-shot learners
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020 · 1901
Earlier work this paper cites.
Federated learning of out-of-vocabulary words
Mingqing Chen, Rajiv Mathews, Tom Ouyang, and Françoise Beaufays. 2019 · 1903
Earlier work this paper cites.
Long short-term memory
Sepp Hochreiter and Jürgen Schmidhuber. 1997 · 1997
Earlier work this paper cites.
Training keyword spotting models on non-iid data with federated learning
Andrew Hard, Kurt Partridge, Cameron Nguyen, Niranjan Subrahmanya, Aishanee Shah, Pai Zhu, Ignacio Lopez Moreno, and Rajiv Mathews. 2020 · 2005
Earlier work this paper cites.
Calibrating Noise to Sensitivity in Private Data Analysis , pages 265–284. Springer Berlin Heidelberg
Cynthia Dwork, Frank McSherry, Kobbi Nissim, and Adam Smith. 2006 · 2006
Earlier work this paper cites.
Random features for large-scale kernel machines
Ali Rahimi and Benjamin Recht. 2007 · 2007
Earlier work this paper cites.
Training production language models without memorizing user data
Swaroop Ramaswamy, Om Thakkar, Rajiv Mathews, Galen Andrew, H. Brendan McMahan, and Françoise Beaufays. 2020 · 2009
Earlier work this paper cites.
Differential privacy in new settings
Cynthia Dwork. 2010 · 2010
Earlier work this paper cites.
A firm foundation for private data analysis
Cynthia Dwork. 2011 · 2011
Earlier work this paper cites.
Japanese and korean voice search
Mike Schuster and Kaisuke Nakajima. 2012 · 2012
Earlier work this paper cites.
The algorithmic foundations of differential privacy
Cynthia Dwork and Aaron Roth. 2014 · 2014
Earlier work this paper cites.
Deep learning with differential privacy
Martín Abadi, Andy Chu, Ian J. Goodfellow, H. B. McMahan, Ilya Mironov, Kunal Talwar, and Li Zhang. 2016 · 2016
Earlier work this paper cites.
Neural machine translation of rare words with subword units
Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016 · 2016
Earlier work this paper cites.
Communication-efficient learning of deep networks from decentralized data
H. B. McMahan, Eider Moore, D. Ramage, S. Hampson, and B. A. Y. Arcas. 2017 · 2017
Earlier work this paper cites.
Attention is all you need
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin. 2017 · 2017
Earlier work this paper cites.
Federated learning for mobile keyboard prediction
Andrew Hard, Chloé M Kiddon, Daniel Ramage, Francoise Beaufays, Hubert Eichner, Kanishka Rao, Rajiv Mathews, and Sean Augenstein. 2018 · 2018
Earlier work this paper cites.
Sentencepiece: A simple and language independent subword tokenizer and detokenizer for neural text processing
Taku Kudo and John Richardson. 2018 · 2018
Earlier work this paper cites.
Learning differentially private recurrent language models
Brendan McMahan, Daniel Ramage, Kunal Talwar, and Li Zhang. 2018 · 2018
Cited alongside, same era.
Tensorflow federated stack overflow dataset
The TensorFlow Federated Authors. 2019 · 2019
Cited alongside, same era.
Automated scalable bayesian inference via hilbert coresets
Trevor Campbell and Tamara Broderick. 2019 · 2019
Cited alongside, same era.
BERT: pre-training of deep bidirectional transformers for language understanding
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019 · 2019
Cited alongside, same era.
Tinybert: Distilling bert for natural language understanding
Xiaoqi Jiao, Yichun Yin, Lifeng Shang, Xin Jiang, Xiao Chen, Linlin Li, F. Wang, and Qun Liu. 2019 · 2019
Cited alongside, same era.
Advances and open problems in federated learning
P. Kairouz, H. B. McMahan, Brendan Avent, A. Bellet, M. Bennis, A. Bhagoji, Keith Bonawitz, Zachary B. Charles, Graham Cormode, Rachel Cummings, Rafael G. L. D’Oliveira, S. Rouayheb, David Evans, Josh Gardner, Zachary Garrett, Adrià Gascón, Badih Ghazi, Phillip B. Gibbons, M. Gruteser, Z. Harchaoui, Chaoyang He, Lie He, Zhouyuan Huo, Ben Hutchinson, Justin Hsu, Martin Jaggi, T. Javidi, Gauri Joshi, M. Khodak, Jakub Konecný, A. Korolova, F. Koushanfar, O. Koyejo, Tancrède Lepoint, Yang Liu, Prateek Mittal, M. Mohri, R. Nock, A. Özgür, R. Pagh, Mariana Raykova, Hang Qi, D. Ramage, R. Raskar, D. Song, Weikang Song, S. Stich, Ziteng Sun, A. Suresh, Florian Tramèr, Praneeth Vepakomma, Jianyu Wang, Li Xiong, Zheng Xu, Qiang Yang, Felix X. Yu, Han Yu, and Sen Zhao. 2019 · 2019
Large language models can be strong differentially private learners
Xuechen Li, Florian Tramèr, Percy Liang, and Tatsunori B. Hashimoto. 2021 · 2021
Later among the works it cites.
Adaptive federated optimization
Sashank J. Reddi, Zachary Charles, Manzil Zaheer, Zachary Garrett, Keith Rush, Jakub Konečný, Sanjiv Kumar, and Hugh Brendan McMahan. 2021 · 2021
Later among the works it cites.
A field guide to federated optimization
Jianyu Wang, Zachary Charles, Zheng Xu, Gauri Joshi, H Brendan McMahan, Blaise Aguera y Arcas, Maruan Al-Shedivat, Galen Andrew, Salman Avestimehr, Katharine Daly, et al. 2021 · 2021
Later among the works it cites.
Bayesian coresets: Revisiting the nonconvex optimization perspective
Jacky Zhang, Rajiv Khanna, Anastasios Kyrillidis, and Sanmi Koyejo. 2021 · 2021
Later among the works it cites.
Training a tokenizer for free with private federated learning
Eugene Bagdasaryan, Congzheng Song, Rogier van Dalen, Matt Seigel, and Áine Cahill. 2022 · 2022
Later among the works it cites.
alphaXiv searches the wider corpus for related work and actual follow-ups.
alphaXiv is searching for related work…
Cited alongside, same era.
Language models are unsupervised multitask learners
Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019 · 2019
Cited alongside, same era.
Depth-adaptive transformer
Maha Elbayad, Jiatao Gu, Edouard Grave, and Michael Auli. 2020 · 2020
Cited alongside, same era.
Compressing BERT: Studying the effects of weight pruning on transfer learning
Mitchell Gordon, Kevin Duh, and Nicholas Andrews. 2020 · 2020
Cited alongside, same era.
Differentially private language models benefit from public pre-training
Gavin Kerrigan, Dylan Slack, and Jens Tuyls. 2020 · 2020
Cited alongside, same era.
Exploring the limits of transfer learning with a unified text-to-text transformer
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020 · 2020
Cited alongside, same era.
Mobilebert: a compact task-agnostic bert for resource-limited devices
Zhiqing Sun, Hongkun Yu, Xiaodan Song, Renjie Liu, Yiming Yang, and Denny Zhou. 2020 · 2020
Cited alongside, same era.
Differentially private bias-term only fine-tuning of foundation models
Zhiqi Bu, Yu-Xiang Wang, Sheng Zha, and George Karypis. 2022 · 2022
Later among the works it cites.
Federated select: A primitive for communication-and memory-efficient federated learning
Zachary Charles, Kallista Bonawitz, Stanislav Chiknavaryan, Brendan McMahan, et al. 2022 · 2022
Later among the works it cites.
Private adaptive optimization with side information
Tian Li, M. Zaheer, Sashank J. Reddi, and Virginia Smith. 2022 · 2022
Later among the works it cites.
Where to begin? on the impact of pre-training and initialization in federated learning
John Nguyen, Jianyu Wang, Kshitiz Malik, Maziar Sanjabi, and Michael Rabbat. 2022 · 2022
Later among the works it cites.
Training text-to-text transformers with privacy guarantees
Natalia Ponomareva, Jasmijn Bastings, and Sergei Vassilvitskii. 2022 · 2022
Later among the works it cites.
Lamda: Language models for dialog applications
Romal Thoppilan, Daniel De Freitas, Jamie Hall, Noam Shazeer, Apoorv Kulshreshtha, Heng-Tze Cheng, Alicia Jin, Taylor Bos, Leslie Baker, Yu Du, YaGuang Li, Hongrae Lee, Huaixiu Steven Zheng, Amin Ghafouri, Marcelo Menegali, Yanping Huang, Maxim Krikun, Dmitry Lepikhin, James Qin, Dehao Chen, Yuanzhong Xu, Zhifeng Chen, Adam Roberts, Maarten Bosma, Vincent Zhao, Yanqi Zhou, Chung-Ching Chang, Igor Krivokon, Will Rusch, Marc Pickett, Pranesh Srinivasan, Laichee Man, Kathleen Meier-Hellstern, Meredith Ringel Morris, Tulsee Doshi, Renelito Delos Santos, Toju Duke, Johnny Soraker, Ben Zevenbergen, Vinodkumar Prabhakaran, Mark Diaz, Ben Hutchinson, Kristen Olson, Alejandra Molina, Erin Hoffman-John, Josh Lee, Lora Aroyo, Ravi Rajakumar, Alena Butryna, Matthew Lamm, Viktoriya Kuzmina, Joe Fenton, Aaron Cohen, Rachel Bernstein, Ray Kurzweil, Blaise Aguera-Arcas, Claire Cui, Marian Croak, Ed Chi, and Quoc Le. 2022 · 2022
Later among the works it cites.
Motley: Benchmarking heterogeneity and personalization in federated learning
Shanshan Wu, Tian Li, Zachary Charles, Yu Xiao, Ziyu Liu, Zheng Xu, and Virginia Smith. 2022 · 2022
Later among the works it cites.
Differentially private fine-tuning of language models
Da Yu, Saurabh Naik, Arturs Backurs, Sivakanth Gopi, Huseyin A. Inan, Gautam Kamath, Janardhan Kulkarni, Yin Tat Lee, Andre Manoel, Lukas Wutschitz, Sergey Yekhanin, and Huishuai Zhang. 2022 · 2022
Later among the works it cites.
Why is public pretraining necessary for private model training?
Arun Ganesh, Mahdi Haghifam, Milad Nasr, Sewoong Oh, Thomas Steinke, Om Thakkar, Abhradeep Thakurta, and Lun Wang. 2023 · 2023
Closest in time.
Federated domain adaptation via gradient projection
Enyi Jiang, Yibo Jacky Zhang, and Oluwasanmi Koyejo. 2023 · 2023
Closest in time.
Federated learning of gboard language models with differential privacy
Zheng Xu, Yanxiang Zhang, Galen Andrew, Christopher Choquette, Peter Kairouz, Brendan McMahan, Jesse Rosenstock, and Yuanbo Zhang. 2023 · 2023
Closest in time.