Fetching the paper…
Reading the bibliography…
We introduce Dataset Grouper, a library to create large-scale group-structured (e.g., federated) datasets, enabling federated learning simulation at the scale of foundation models.
Calibrating Noise to Sensitivity in Private Data Analysis
Cynthia Dwork, Frank McSherry, Kobbi Nissim, and Adam D. Smith · 2006
Earlier work this paper cites.
Zipf’s word frequency law in natural language: A critical review and future directions
Steven T Piantadosi · 2014
Earlier work this paper cites.
TensorFlow: Large-Scale Machine Learning on Heterogeneous Systems, 2015
Martín Abadi, Ashish Agarwal, Paul Barham, Eugene Brevdo, Zhifeng Chen, Craig Citro, Greg S. Corrado, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Ian Goodfellow, Andrew Harp, Geoffrey Irving, Michael Isard, Yangqing Jia, Rafal Jozefowicz, Lukasz Kaiser, Manjunath Kudlur, Josh Levenberg, Dandelion Mané, Rajat Monga, Sherry Moore, Derek Murray, Chris Olah, Mike Schuster, Jonathon Shlens, Benoit Steiner, Ilya Sutskever, Kunal Talwar, Paul Tucker, Vincent Vanhoucke, Vijay Vasudevan, Fernanda Viégas, Oriol Vinyals, Pete Warden, Martin Wattenberg, Martin Wicke, Yuan Yu, and Xiaoqiang Zheng · 2015
Earlier work this paper cites.
Aligning Books and Movies: Towards Story-Like Visual Explanations by Watching Movies and Reading Books
Yukun Zhu, Ryan Kiros, Rich Zemel, Ruslan Salakhutdinov, Raquel Urtasun, Antonio Torralba, and Sanja Fidler · 2015
Earlier work this paper cites.
Deep Learning with Differential Privacy
Martín Abadi, Andy Chu, Ian J. Goodfellow, H. Brendan McMahan, Ilya Mironov, Kunal Talwar, and Li Zhang · 2016
Earlier work this paper cites.
Human behavior and the principle of least effort: An introduction to human ecology
George Kingsley Zipf · 2016
Earlier work this paper cites.
Google’s Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V. Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, Jeff Klingner, Apurva Shah, Melvin Johnson, Xiaobing Liu, Lukasz Kaiser, Stephan Gouws, Yoshikiyo Kato, Taku Kudo, Hideto Kazawa, Keith Stevens, George Kurian, Nishant Patil, Wei Wang, Cliff Young, Jason Smith, Jason Riesa, Alex Rudnick, Oriol Vinyals, Greg Corrado, Macduff Hughes, and Jeffrey Dean · 2016
Earlier work this paper cites.
Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks
Chelsea Finn, Pieter Abbeel, and Sergey Levine · 2017
Earlier work this paper cites.
Communication-Efficient Learning of Deep Networks from Decentralized Data
Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Arcas · 2017
Earlier work this paper cites.
Letter value plots: Boxplots for large data
Heike Hofmann, Hadley Wickham, and Karen Kafadar · 2017
Earlier work this paper cites.
LEAF: A benchmark for federated settings
Sebastian Caldas, Sai Meher Karthik Duddu, Peter Wu, Tian Li, Jakub Konečný, H. Brendan McMahan, Virginia Smith, and Ameet Talwalkar · 2018
Earlier work this paper cites.
Learning Differentially Private Recurrent Language Models
H. Brendan McMahan, Daniel Ramage, Kunal Talwar, and Li Zhang · 2018
Earlier work this paper cites.
JAX: composable transformations of Python+NumPy programs, 2018
James Bradbury, Roy Frostig, Peter Hawkins, Matthew James Johnson, Chris Leary, Dougal Maclaurin, George Necula, Adam Paszke, Jake VanderPlas, Skye Wanderman-Milne, and Qiao Zhang · 2018
Earlier work this paper cites.
On First-Order Meta-Learning Algorithms
Alex Nichol and John Schulman · 2018
Earlier work this paper cites.
Local SGD with Periodic Averaging: Tighter Analysis and Adaptive Synchronization
Farzin Haddadpour, Mohammad Mahdi Kamani, Mehrdad Mahdavi, and Viveck Cadambe · 2019
Earlier work this paper cites.
Communication Trade-offs for Local-SGD with Large Step Size
Aymeric Dieuleveut and Kumar Kshitij Patel · 2019
Earlier work this paper cites.
OpenWebText Corpus
Ellie Pavlick Stefanie Tellex Aaron Gokaslan, Vanya Cohen · 2019
Earlier work this paper cites.
Measuring the Effects of Non-Identical Data Distribution for Federated Visual Classification
Tzu-Ming Harry Hsu, Hang Qi, and Matthew Brown · 2019
Earlier work this paper cites.
PyTorch: An imperative style, high-performance deep learning library
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al · 2019
Earlier work this paper cites.
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova · 2019
Earlier work this paper cites.
Improving Federated Learning Personalization via Model Agnostic Meta Learning
Yihan Jiang, Jakub Konečnỳ, Keith Rush, and Sreeram Kannan · 2019
Earlier work this paper cites.
FedML: A Research Library and Benchmark for Federated Machine Learning
Chaoyang He, Songze Li, Jinhyun So, Mi Zhang, Hongyi Wang, Xiaoyang Wang, Praneeth Vepakomma, Abhishek Singh, Hang Qiu, Li Shen, Peilin Zhao, Yan Kang, Yang Liu, Ramesh Raskar, Qiang Yang, Murali Annavaram, and Salman Avestimehr · 2020
Earlier work this paper cites.
Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu · 2020
Earlier work this paper cites.
Flower: A Friendly Federated Learning Research Framework
Daniel J Beutel, Taner Topal, Akhil Mathur, Xinchi Qiu, Javier Fernandez-Marques, Yan Gao, Lorenzo Sani, Hei Li Kwing, Titouan Parcollet, Pedro PB de Gusmão, and Nicholas D Lane · 2020
Earlier work this paper cites.
Tighter Theory for Local SGD on Identical and Heterogeneous Data
A Khaled, K Mishchenko, and P Richtárik · 2020
Earlier work this paper cites.
SCAFFOLD: stochastic controlled averaging for federated learning
Sai Praneeth Karimireddy, Satyen Kale, Mehryar Mohri, Sashank J. Reddi, Sebastian U. Stich, and Ananda Theertha Suresh · 2020
Earlier work this paper cites.
Tackling the Objective Inconsistency Problem in Heterogeneous Federated Optimization
Jianyu Wang, Qinghua Liu, Hao Liang, Gauri Joshi, and H. Vincent Poor · 2020
Earlier work this paper cites.
Federated Learning With Differential Privacy: Algorithms and Performance Analysis
Kang Wei, Jun Li, Ming Ding, Chuan Ma, Howard H. Yang, Farhad Farokhi, Shi Jin, Tony Q. S. Quek, and H. Vincent Poor · 2020
Cited alongside, same era.
Device Heterogeneity in Federated Learning: A Superquantile Approach
Yassine Laguel, Krishna Pillutla, Jérôme Malick, and Zaid Harchaoui · 2020
Cited alongside, same era.
Distributionally Robust Federated Averaging
Yuyang Deng, Mohammad Mahdi Kamani, and Mehrdad Mahdavi · 2020
Cited alongside, same era.
Robust Federated Learning: The Case of Affine Distribution Shifts
Amirhossein Reisizadeh, Farzan Farnia, Ramtin Pedarsani, and Ali Jadbabaie · 2020
Cited alongside, same era.
Personalized Federated Learning with Moreau Envelopes
Canh T. Dinh, Nguyen Tran, and Josh Nguyen · 2020
Cited alongside, same era.
The Pile: An 800GB Dataset of Diverse Text for Language Modeling
Fine-tuning is Fine in Federated Learning
Gary Cheng, Karan Chadha, and John Duchi · 2021
Later among the works it cites.
Reduced, Reused and Recycled: The Life of a Dataset in Machine Learning Research
Bernard Koch, Emily Denton, Alex Hanna, and Jacob G. Foster · 2021
Later among the works it cites.
tf.data: A machine learning data processing framework
Derek G Murray, Jiri Simsa, Ana Klimovic, and Ihor Indyk · 2021
Later among the works it cites.
What Does it Mean for a Language Model to Preserve Privacy?
Hannah Brown, Katherine Lee, Fatemehsadat Mireshghallah, Reza Shokri, and Florian Tramèr · 2022
Later among the works it cites.
FedNLP: Benchmarking Federated Learning Methods for Natural Language Processing Tasks
Bill Yuchen Lin, Chaoyang He, Zihang Ze, Hulin Wang, Yufen Hua, Christophe Dupuy, Rahul Gupta, Mahdi Soltanolkotabi, Xiang Ren, and Salman Avestimehr · 2022
Later among the works it cites.
FedScale: Benchmarking Model and System Performance of Federated Learning at Scale
alphaXiv searches the wider corpus for related work and actual follow-ups.
alphaXiv is searching for related work…
Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, Shawn Presser, and Connor Leahy · 2020
Cited alongside, same era.
Scaling Laws for Neural Language Models
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei · 2020
Cited alongside, same era.
Big Bird: Transformers for Longer Sequences
Manzil Zaheer, Guru Guruganesh, Kumar Avinava Dubey, Joshua Ainslie, Chris Alberti, Santiago Ontanon, Philip Pham, Anirudh Ravula, Qifan Wang, Li Yang, et al · 2020
Cited alongside, same era.
Longformer: The Long-Document Transformer
Iz Beltagy, Matthew E. Peters, and Arman Cohan · 2020
Cited alongside, same era.
Array programming with NumPy
Charles R. Harris, K. Jarrod Millman, Stéfan J van der Walt, Ralf Gommers, Pauli Virtanen, David Cournapeau, Eric Wieser, Julian Taylor, Sebastian Berg, Nathaniel J. Smith, Robert Kern, Matti Picus, Stephan Hoyer, Marten H. van Kerkwijk, Matthew Brett, Allan Haldane, Jaime Fernández del Río, Mark Wiebe, Pearu Peterson, Pierre Gérard-Marchant, Kevin Sheppard, Tyler Reddy, Warren Weckesser, Hameer Abbasi, Christoph Gohlke, and Travis E. Oliphant · 2020
Cited alongside, same era.
Personalized Federated Learning with Theoretical Guarantees: A Model-Agnostic Meta-Learning Approach
Alireza Fallah, Aryan Mokhtari, and Asuman E. Ozdaglar · 2020
Cited alongside, same era.
The Power of Scale for Parameter-Efficient Prompt Tuning
Brian Lester, Rami Al-Rfou, and Noah Constant · 2021
Cited alongside, same era.
Fan Lai, Yinwei Dai, Sanjay Sri Vallabh Singapuram, Jiachen Liu, Xiangfeng Zhu, Harsha V. Madhyastha, and Mosharaf Chowdhury · 2022
Later among the works it cites.
Improved Differential Privacy for SGD via Optimal Private Linear Operators on Adaptive Streams
Sergey Denisov, H Brendan McMahan, John Rush, Adam Smith, and Abhradeep Guha Thakurta · 2022
Later among the works it cites.
Byzantine-Robust Learning on Heterogeneous Datasets via Bucketing
Sai Praneeth Karimireddy, Lie He, and Martin Jaggi · 2022
Later among the works it cites.
Robustness and Personalization in Federated Learning: A Unified Approach via Regularization
Achintya Kundu, Pengqian Yu, Laura Wynter, and Shiau Hong Lim · 2022
Later among the works it cites.
Back to the drawing board: A critical evaluation of poisoning attacks on production federated learning
Virat Shejwalkar, Amir Houmansadr, Peter Kairouz, and Daniel Ramage · 2022
Later among the works it cites.
Personalization Improves Privacy-Accuracy Tradeoffs in Federated Learning
Alberto Bietti, Chen-Yu Wei, Miroslav Dudík, John Langford, and Zhiwei Steven Wu · 2022
Later among the works it cites.
FLAIR: Federated Learning Annotated Image Repository
Congzheng Song, Filip Granqvist, and Kunal Talwar · 2022
Later among the works it cites.
FLamby: Datasets and Benchmarks for Cross-Silo Federated Learning in Realistic Healthcare Settings
Jean Ogier du Terrail, Samy-Safwan Ayed, Edwige Cyffers, Felix Grimberg, Chaoyang He, Regis Loeb, Paul Mangold, Tanguy Marchand, Othmane Marfoq, Erum Mushtaq, Boris Muzellec, Constantin Philippenko, Santiago Silva, Maria Telenczuk, Shadi Albarqouni, Salman Avestimehr, Aurélien Bellet, Aymeric Dieuleveut, Martin Jaggi, Sai Praneeth Karimireddy, Marco Lorenzi, Giovanni Neglia, Marc Tommasi, and Mathieu Andreux · 2022
Later among the works it cites.
Motley: Benchmarking Heterogeneity and Personalization in Federated Learning
Shanshan Wu, Tian Li, Zachary Charles, Yu Xiao, Ziyu Liu, Zheng Xu, and Virginia Smith · 2022
Later among the works it cites.
pFL-Bench: A Comprehensive Benchmark for Personalized Federated Learning
Daoyuan Chen, Dawei Gao, Weirui Kuang, Yaliang Li, and Bolin Ding · 2022
Later among the works it cites.
Scaling Language Model Size in Cross-Device Federated Learning
Jae Hun Ro, Theresa Breiner, Lara McConnaughey, Mingqing Chen, Ananda Theertha Suresh, Shankar Kumar, and Rajiv Mathews · 2022
Later among the works it cites.
Extending the WILDS Benchmark for Unsupervised Adaptation
Shiori Sagawa, Pang Wei Koh, Tony Lee, Irena Gao, Sang Michael Xie, Kendrick Shen, Ananya Kumar, Weihua Hu, Michihiro Yasunaga, Henrik Marklund, Sara Beery, Etienne David, Ian Stavness, Wei Guo, Jure Leskovec, Kate Saenko, Tatsunori Hashimoto, Sergey Levine, Chelsea Finn, and Percy Liang · 2022
Later among the works it cites.
Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation
Ofir Press, Noah A. Smith, and Mike Lewis · 2022
Later among the works it cites.
PromptFL: Let Federated Participants Cooperatively Learn Prompts Instead of Models–Federated Learning in Age of Foundation Model
Tao Guo, Song Guo, Junxiao Wang, and Wenchao Xu · 2022
Later among the works it cites.
Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed H. Chi, Quoc V. Le, and Denny Zhou · 2022
Later among the works it cites.
Iterated vector fields and conservatism, with applications to federated learning
Zachary Charles and Keith Rush · 2022
Later among the works it cites.
FedAvg with Fine Tuning: Local Updates Lead to Representation Learning
Liam Collins, Hamed Hassani, Aryan Mokhtari, and Sanjay Shakkottai · 2022
Later among the works it cites.
How to DP-fy ML: A Practical Guide to Machine Learning with Differential Privacy
Natalia Ponomareva, Hussein Hazimeh, Alex Kurakin, Zheng Xu, Carson Denison, H. Brendan McMahan, Sergei Vassilvitskii, Steve Chien, and Abhradeep Guha Thakurta · 2023
Closest in time.
Federated learning with superquantile aggregation for heterogeneous data
Krishna Pillutla, Yassine Laguel, Jérôme Malick, and Zaid Harchaoui · 2023
Closest in time.
https://github.com/p-lambda/wilds/issues/73
How do I access data from only one group? Github Issue #73 for p-lambda/wilds · 2023
Closest in time.
Federated Prompting and Chain-of-Thought Reasoning for Improving LLMs Answering
Xiangyang Liu, Tianqi Pang, and Chenyou Fan · 2023
Closest in time.
Towards Building the Federated GPT: Federated Instruction Tuning
Jianyi Zhang, Saeed Vahidian, Martin Kuo, Chunyuan Li, Ruiyi Zhang, Guoyin Wang, and Yiran Chen · 2023
Closest in time.