Fetching the paper…
Reading the bibliography…
Pre-training is prevalent in nowadays deep learning to improve the learned model's performance.
The fractal geometry of nature , volume 1
Benoit B Mandelbrot and Benoit B Mandelbrot · 1982
Earlier work this paper cites.
Improved baselines with momentum contrastive learning
Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming He · 2003
Earlier work this paper cites.
Mime: Mimicking centralized stochastic algorithms in federated learning
Sai Praneeth Karimireddy, Martin Jaggi, Satyen Kale, Mehryar Mohri, Sashank J Reddi, Sebastian U Stich, and Ananda Theertha Suresh · 2008
Earlier work this paper cites.
Imagenet: A large-scale hierarchical image database
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei · 2009
Earlier work this paper cites.
Learning multiple layers of features from tiny images
Alex Krizhevsky, Geoffrey Hinton, et al · 2009
Earlier work this paper cites.
Why does unsupervised pre-training help deep learning?
Dumitru Erhan, Aaron Courville, Yoshua Bengio, and Pascal Vincent · 2010
Earlier work this paper cites.
Fractals everywhere
Michael F Barnsley · 2014
Earlier work this paper cites.
The cityscapes dataset
Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Scharwächter, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele · 2015
Earlier work this paper cites.
Adam: A method for stochastic optimization
Diederik P Kingma and Jimmy Ba · 2015
Earlier work this paper cites.
Tiny imagenet visual recognition challenge
Ya Le and Xuan Yang · 2015
Earlier work this paper cites.
Imagenet large scale visual recognition challenge
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al · 2015
Earlier work this paper cites.
The cityscapes dataset for semantic urban scene understanding
Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele · 2016
Earlier work this paper cites.
Overcoming catastrophic forgetting in neural networks
James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska, et al · 2017
Earlier work this paper cites.
Communication-efficient learning of deep networks from decentralized data
H Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, et al · 2017
Earlier work this paper cites.
Revisiting unreasonable effectiveness of data in deep learning era
Chen Sun, Abhinav Shrivastava, Saurabh Singh, and Abhinav Gupta · 2017
Earlier work this paper cites.
The devil is in the tails: Fine-grained classification in the wild
Grant Van Horn and Pietro Perona · 2017
Earlier work this paper cites.
Places: A 10 million image database for scene recognition
Bolei Zhou, Agata Lapedriza, Aditya Khosla, Aude Oliva, and Antonio Torralba · 2017
Earlier work this paper cites.
Fan Zhou and Guojing Cong · 2017
Earlier work this paper cites.
Leaf: A benchmark for federated settings
Sebastian Caldas, Peter Wu, Tian Li, Jakub Konečnỳ, H Brendan McMahan, Virginia Smith, and Ameet Talwalkar · 2018
Earlier work this paper cites.
Encoder-decoder with atrous separable convolution for semantic image segmentation
Liang-Chieh Chen, Yukun Zhu, George Papandreou, Florian Schroff, and Hartwig Adam · 2018
Earlier work this paper cites.
Bert: Pre-training of deep bidirectional transformers for language understanding
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova · 2018
Earlier work this paper cites.
Identifying medical diagnoses and treatable diseases by image-based deep learning
Daniel S Kermany, Michael Goldbaum, Wenjia Cai, Carolina CS Valentim, Huiying Liang, Sally L Baxter, Alex McKeown, Ge Yang, Xiaokang Wu, Fangbing Yan, et al · 2018
Earlier work this paper cites.
Visualizing the loss landscape of neural nets
Hao Li, Zheng Xu, Gavin Taylor, Christoph Studer, and Tom Goldstein · 2018
Earlier work this paper cites.
Mobilenetv2: Inverted residuals and linear bottlenecks
Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chieh Chen · 2018
Earlier work this paper cites.
The inaturalist species classification and detection dataset
Grant Van Horn, Oisin Mac Aodha, Yang Song, Yin Cui, Chen Sun, Alex Shepard, Hartwig Adam, Pietro Perona, and Serge Belongie · 2018
Earlier work this paper cites.
Unsupervised feature learning via non-parametric instance discrimination
Zhirong Wu, Yuanjun Xiong, Stella X Yu, and Dahua Lin · 2018
Earlier work this paper cites.
Federated learning with non-iid data
Yue Zhao, Meng Li, Liangzhen Lai, Naveen Suda, Damon Civin, and Vikas Chandra · 2018
Earlier work this paper cites.
On the convergence of local descent methods in federated learning
Farzin Haddadpour and Mehrdad Mahdavi · 2019
Cited alongside, same era.
Visualizing and understanding the effectiveness of bert
Yaru Hao, Li Dong, Furu Wei, and Ke Xu · 2019
Cited alongside, same era.
Rethinking imagenet pre-training
Kaiming He, Ross Girshick, and Piotr Dollár · 2019
Cited alongside, same era.
Using pre-training can improve model robustness and uncertainty
Dan Hendrycks, Kimin Lee, and Mantas Mazeika · 2019
Cited alongside, same era.
Measuring the effects of non-identical data distribution for federated visual classification
Tzu-Ming Harry Hsu, Hang Qi, and Matthew Brown · 2019
Cited alongside, same era.
Fedbe: Making bayesian model ensemble applicable to federated learning
Hong-You Chen and Wei-Lun Chao · 2021
Later among the works it cites.
Exploring simple siamese representation learning
Xinlei Chen and Kaiming He · 2021
Later among the works it cites.
Fine-tuning is fine in federated learning
Gary Cheng, Karan Chadha, and John Duchi · 2021
Later among the works it cites.
On robustness and transferability of convolutional neural networks
Josip Djolonga, Jessica Yung, Michael Tschannen, Rob Romijnders, Lucas Beyer, Alexander Kolesnikov, Joan Puigcerver, Matthias Minderer, Alexander D’Amour, Dan Moldovan, et al · 2021
Later among the works it cites.
How well do self-supervised models transfer?
Linus Ericsson, Henry Gouk, and Timothy M Hospedales · 2021
Later among the works it cites.
Synthasr: Unlocking synthetic data for speech recognition
alphaXiv searches the wider corpus for related work and actual follow-ups.
alphaXiv is searching for related work…
Peter Kairouz, H Brendan McMahan, Brendan Avent, Aurélien Bellet, Mehdi Bennis, Arjun Nitin Bhagoji, Keith Bonawitz, Zachary Charles, Graham Cormode, Rachel Cummings, et al · 2019
Cited alongside, same era.
A style-based generator architecture for generative adversarial networks
Tero Karras, Samuli Laine, and Timo Aila · 2019
Cited alongside, same era.
Do better imagenet models transfer better?
Simon Kornblith, Jonathon Shlens, and Quoc V Le · 2019
Cited alongside, same era.
Feddane: A federated newton-type method
Tian Li, Anit Kumar Sahu, Manzil Zaheer, Maziar Sanjabi, Ameet Talwalkar, and Virginia Smithy · 2019
Cited alongside, same era.
Local sgd converges fast and communicates little
Sebastian U Stich · 2019
Cited alongside, same era.
Xlnet: Generalized autoregressive pretraining for language understanding
Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Russ R Salakhutdinov, and Quoc V Le · 2019
Cited alongside, same era.
Federated learning with unbiased gradient aggregation and controllable meta updating
Xin Yao, Tianchi Huang, Rui-Xiao Zhang, Ruiyu Li, and Lifeng Sun · 2019
Cited alongside, same era.
Amin Fazel, Wei Yang, Yulan Liu, Roberto Barra-Chicote, Yixiong Meng, Roland Maas, and Jasha Droppo · 2021
Later among the works it cites.
Self-supervised training enhances online continual learning
Jhair Gallardo, Tyler L Hayes, and Christopher Kanan · 2021
Later among the works it cites.
Self-supervised pretraining of visual features in the wild
Priya Goyal, Mathilde Caron, Benjamin Lefaudeux, Min Xu, Pengchao Wang, Vivek Pai, Mannat Singh, Vitaliy Liptchinsky, Ishan Misra, Armand Joulin, et al · 2021
Later among the works it cites.
Self-supervised learning: Generative or contrastive
Xiao Liu, Fanjin Zhang, Zhenyu Hou, Li Mian, Zhaoyu Wang, Jing Zhang, and Jie Tang · 2021
Later among the works it cites.
An empirical investigation of the role of pre-training in lifelong learning
Sanket Vaibhav Mehta, Darshan Patil, Sarath Chandar, and Emma Strubell · 2021
Later among the works it cites.
Rethinking architecture design for tackling data heterogeneity in federated learning
Liangqiong Qu, Yuyin Zhou, Paul Pu Liang, Yingda Xia, Feifei Wang, Li Fei-Fei, Ehsan Adeli, and Daniel Rubin · 2021
Later among the works it cites.
Learning transferable visual models from natural language supervision
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever · 2021
Later among the works it cites.
Adaptive federated optimization
Sashank Reddi, Zachary Charles, Manzil Zaheer, Zachary Garrett, Keith Rush, Jakub Konečnỳ, Sanjiv Kumar, and H Brendan McMahan · 2021
Later among the works it cites.
Selfaugment: Automatic augmentation policies for self-supervised learning
Colorado J Reed, Sean Metzger, Aravind Srinivas, Trevor Darrell, and Kurt Keutzer · 2021
Later among the works it cites.
Pretraining federated text models for next word prediction
Joel Stremmel and Arjun Singh · 2021
Later among the works it cites.
Initialize with mask: For more efficient federated learning
Zirui Zhu and Lifeng Sun · 2021
Later among the works it cites.
Improving fractal pre-training
Connor Anderson and Ryan Farrell · 2022
Closest in time.
Fedsyn: Synthetic data generation using federated learning
Monik Raj Behera, Sudhir Upadhyay, Suresh Shetty, Sudha Priyadarshini, Palka Patel, and Ker Farn Lee · 2022
Closest in time.
OmniTab: Pretraining with natural and synthetic data for few-shot table-based question answering
Zhengbao Jiang, Yi Mao, Pengcheng He, Graham Neubig, and Weizhu Chen · 2022
Closest in time.
Fednlp: Benchmarking federated learning methods for natural language processing tasks
Bill Yuchen Lin, Chaoyang He, Zihang Zeng, Hulin Wang, Yufen Huang, Mahdi Soltanolkotabi, Xiang Ren, and Salman Avestimehr · 2022
Closest in time.
Orchestra: Unsupervised federated learning via globally consistent clustering
Ekdeep Singh Lubana, Chi Ian Tang, Fahim Kawsar, Robert P Dick, and Akhil Mathur · 2022
Closest in time.
Where to begin? exploring the impact of pre-training and initialization in federated learning
John Nguyen, Kshitiz Malik, Maziar Sanjabi, and Michael Rabbat · 2022
Closest in time.
Self-supervised pretraining improves self-supervised pretraining
Colorado J Reed, Xiangyu Yue, Ani Nrusimha, Sayna Ebrahimi, Vivek Vijaykumar, Richard Mao, Bo Li, Shanghang Zhang, Devin Guillory, Sean Metzger, et al · 2022
Closest in time.
Federated learning from pre-trained models: A contrastive learning approach
Yue Tan, Guodong Long, Jie Ma, Lu Liu, Tianyi Zhou, and Jing Jiang · 2022
Closest in time.
Pretrained models for multilingual federated learning
Orion Weller, Marc Marone, Vladimir Braverman, Dawn Lawrie, and Benjamin Van Durme · 2022
Closest in time.
Insights into pre-training via simpler synthetic tasks
Yuhuai Wu, Felix Li, and Percy Liang · 2022
Closest in time.