Fetching the paper…
Reading the bibliography…
Differentially Private (DP) learning has seen limited success for building large deep learning models of text, and straightforward attempts at applying Differentially Private Stochastic Gradient Descent (DP-SGD) to NLP tasks have resulted in large performance drops and high computational overhead.
A taxonomy of privacy
Daniel J Solove · 2005
Earlier work this paper cites.
Calibrating noise to sensitivity in private data analysis
Cynthia Dwork, Frank McSherry, Kobbi Nissim, and Adam Smith · 2006
Earlier work this paper cites.
Texthide: Tackling data privacy in language understanding tasks
Yangsibo Huang, Zhao Song, Danqi Chen, Kai Li, and Sanjeev Arora · 2010
Earlier work this paper cites.
A stability-based validation procedure for differentially private machine learning
Kamalika Chaudhuri and Staal A Vinterbo · 2013
Earlier work this paper cites.
Stochastic gradient descent with differentially private updates
Shuang Song, Kamalika Chaudhuri, and Anand D Sarwate · 2013
Earlier work this paper cites.
Bypassing the ambient dimension: Private sgd with gradient subspace identification
Yingxue Zhou, Zhiwei Steven Wu, and Arindam Banerjee · 2013
Earlier work this paper cites.
Differentially private empirical risk minimization: Efficient algorithms and tight error bounds, 2014
Raef Bassily, Adam Smith, and Abhradeep Thakurta · 2014
Earlier work this paper cites.
The algorithmic foundations of differential privacy
Cynthia Dwork, Aaron Roth, et al · 2014
Earlier work this paper cites.
Adam: A method for stochastic optimization
Diederik P Kingma and Jimmy Ba · 2014
Earlier work this paper cites.
Efficient per-example gradient computations
Ian Goodfellow · 2015
Earlier work this paper cites.
A diversity-promoting objective function for neural conversation models
Jiwei Li, Michel Galley, Chris Brockett, Jianfeng Gao, and Bill Dolan · 2015
Earlier work this paper cites.
The moral character of cryptographic work
Phillip Rogaway · 2015
Earlier work this paper cites.
Deep learning with differential privacy
Martin Abadi, Andy Chu, Ian Goodfellow, H Brendan McMahan, Ilya Mironov, Kunal Talwar, and Li Zhang · 2016
Earlier work this paper cites.
Semi-supervised knowledge transfer for deep learning from private training data
Nicolas Papernot, Martín Abadi, Ulfar Erlingsson, Ian Goodfellow, and Kunal Talwar · 2016
Earlier work this paper cites.
Understanding deep learning requires rethinking generalization
Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals · 2016
Earlier work this paper cites.
Learning differentially private recurrent language models
H Brendan McMahan, Daniel Ramage, Kunal Talwar, and Li Zhang · 2017
Earlier work this paper cites.
Paulius Micikevicius, Sharan Narang, Jonah Alben, Gregory Diamos, Erich Elsen, David Garcia, Boris Ginsburg, Michael Houston, Oleksii Kuchaiev, Ganesh Venkatesh, et al · 2017
Earlier work this paper cites.
Rényi differential privacy
Ilya Mironov · 2017
Earlier work this paper cites.
The e2e dataset: New challenges for end-to-end generation
Jekaterina Novikova, Ondřej Dušek, and Verena Rieser · 2017
Earlier work this paper cites.
Automatic differentiation in pytorch
Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer · 2017
Earlier work this paper cites.
Membership inference attacks against machine learning models
Reza Shokri, Marco Stronati, Congzheng Song, and Vitaly Shmatikov · 2017
Earlier work this paper cites.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin · 2017
Earlier work this paper cites.
Privacy-preserving neural representations of text
Maximin Coavoux, Shashi Narayan, and Shay B Cohen · 2018
Earlier work this paper cites.
Bert: Pre-training of deep bidirectional transformers for language understanding
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova · 2018
Earlier work this paper cites.
Neural approaches to conversational ai
Jianfeng Gao, Michel Galley, and Lihong Li · 2018
Earlier work this paper cites.
Scalable private learning with pate
Nicolas Papernot, Shuang Song, Ilya Mironov, Ananth Raghunathan, Kunal Talwar, and Úlfar Erlingsson · 2018
Earlier work this paper cites.
The multi-genre nli corpus
Adina Williams, Nikita Nangia, and Samuel R Bowman · 2018
Earlier work this paper cites.
Learning neural templates for text generation
Sam Wiseman, Stuart M Shieber, and Alexander M Rush · 2018
Earlier work this paper cites.
Personalizing dialogue agents: I have a dog, do you have pets too?
Saizheng Zhang, Emily Dinan, Jack Urbanek, Arthur Szlam, Douwe Kiela, and Jason Weston · 2018
Earlier work this paper cites.
Towards private synthetic text generation
Rishi Bommasani, Steven Wu, and Xanda Schofield · 2019
Cited alongside, same era.
The secret sharer: Evaluating and testing unintended memorization in neural networks
Nicholas Carlini, Chang Liu, Úlfar Erlingsson, Jernej Kos, and Dawn Song · 2019
Cited alongside, same era.
The second conversational intelligence challenge (convai2)
Emily Dinan, Varvara Logacheva, Valentin Malykh, Alexander Miller, Kurt Shuster, Jack Urbanek, Douwe Kiela, Arthur Szlam, Iulian Serban, Ryan Lowe, et al · 2019
Cited alongside, same era.
Jinshuo Dong, Aaron Roth, and Weijie J Su · 2019
Cited alongside, same era.
Logan: Membership inference attacks against generative models
Jamie Hayes, Luca Melis, George Danezis, and Emiliano De Cristofaro · 2019
Cited alongside, same era.
Dart: Open-domain structured data record to text generation
Linyong Nan, Dragomir Radev, Rui Zhang, Amrit Rau, Abhinand Sivaprasad, Chiachun Hsieh, Xiangru Tang, Aadit Vyas, Neha Verma, Pranav Krishna, et al · 2020
Later among the works it cites.
Marcel Neunhoeffer, Zhiwei Steven Wu, and Cynthia Dwork · 2020
Later among the works it cites.
Adapterfusion: Non-destructive task composition for transfer learning
Jonas Pfeiffer, Aishwarya Kamath, Andreas Rücklé, Kyunghyun Cho, and Iryna Gurevych · 2020
Later among the works it cites.
Training production language models without memorizing user data
Swaroop Ramaswamy, Om Thakkar, Rajiv Mathews, Galen Andrew, H Brendan McMahan, and Françoise Beaufays · 2020
Later among the works it cites.
alphaXiv searches the wider corpus for related work and actual follow-ups.
alphaXiv is searching for related work…
Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi · 2019
Cited alongside, same era.
Parameter-efficient transfer learning for nlp
Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly · 2019
Cited alongside, same era.
Private selection from private candidates
Jingcheng Liu and Kunal Talwar · 2019
Cited alongside, same era.
Roberta: A robustly optimized bert pretraining approach
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov · 2019
Cited alongside, same era.
Towards automatic generation of shareable synthetic clinical notes using neural language models
Oren Melamud and Chaitanya Shivade · 2019
Cited alongside, same era.
R \ \backslash ’enyi differential privacy of the sampled gaussian mechanism
Ilya Mironov, Kunal Talwar, and Li Zhang · 2019
Cited alongside, same era.
Making the shoe fit: Architectures, initializations, and tuning for learning with privacy
Nicolas Papernot, Steve Chien, Shuang Song, Abhradeep Thakurta, and Ulfar Erlingsson · 2019
Cited alongside, same era.
Andreas Rücklé, Gregor Geigle, Max Glockner, Tilman Beck, Jonas Pfeiffer, Nils Reimers, and Iryna Gurevych · 2020
Later among the works it cites.
Enabling fast differentially private sgd via just-in-time compilation and vectorization
Pranav Subramani, Nicholas Vadivelu, and Gautam Kamath · 2020
Later among the works it cites.
Differentially private synthetic medical data generation using convolutional gans
Amirsina Torfi, Edward A Fox, and Chandan K Reddy · 2020
Later among the works it cites.
Differentially private learning needs better features (or much more data)
Florian Tramèr and Dan Boneh · 2020
Later among the works it cites.
Large-scale differentially private bert
Rohan Anil, Badih Ghazi, Vineet Gupta, Ravi Kumar, and Pasin Manurangsi · 2021
Closest in time.
Private adaptive gradient methods for convex optimization
Hilal Asi, John Duchi, Alireza Fallah, Omid Javidbakht, and Kunal Talwar · 2021
Closest in time.
Bitfit: Simple parameter-efficient fine-tuning for transformer-based masked language-models
Elad Ben Zaken, Shauli Ravfogel, and Yoav Goldberg · 2021
Closest in time.
On the opportunities and risks of foundation models
Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, et al · 2021
Closest in time.
Fast and memory efficient differentially private-sgd via jl projections
Zhiqi Bu, Sivakanth Gopi, Janardhan Kulkarni, Yin Tat Lee, Judy Hanwen Shen, and Uthaipon Tantipongpipat · 2021
Closest in time.
An efficient dp-sgd mechanism for large scale nlp models
Christophe Dupuy, Radhika Arava, Rahul Gupta, and Anna Rumshisky · 2021
Closest in time.
The gem benchmark: Natural language generation, its evaluation and metrics
Sebastian Gehrmann, Tosin Adewumi, Karmanya Aggarwal, Pawan Sasanka Ammanamanchi, Aremu Anuoluwapo, Antoine Bosselut, Khyathi Raghavi Chandu, Miruna Clinciu, Dipanjan Das, Kaustubh D Dhole, et al · 2021
Closest in time.
Numerical composition of differential privacy
Sivakanth Gopi, Yin Tat Lee, and Lukas Wutschitz · 2021
Closest in time.
Learning and evaluating a differentially private pre-trained language model
Shlomo Hoory, Amir Feder, Avichai Tendler, Alon Cohen, Sofia Erell, Itay Laish, Hootan Nakhost, Uri Stemmer, Ayelet Benjamini, Avinatan Hassidim, et al · 2021
Closest in time.
Lora: Low-rank adaptation of large language models
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, and Weizhu Chen · 2021
Closest in time.
The power of scale for parameter-efficient prompt tuning
Brian Lester, Rami Al-Rfou, and Noah Constant · 2021
Closest in time.
Prefix-tuning: Optimizing continuous prompts for generation
Xiang Lisa Li and Percy Liang · 2021
Closest in time.
Xiao Liu, Yanan Zheng, Zhengxiao Du, Ming Ding, Yujie Qian, Zhilin Yang, and Jie Tang · 2021
Closest in time.
Compacter: Efficient low-rank hypercomplex adapter layers
Rabeeh Karimi Mahabadi, James Henderson, and Sebastian Ruder · 2021
Closest in time.
Privacy regularization: Joint privacy-utility optimization in language models
Fatemehsadat Mireshghallah, Huseyin A Inan, Marcello Hasegawa, Victor Rühle, Taylor Berg-Kirkpatrick, and Robert Sim · 2021
Closest in time.
Hyperparameter tuning with renyi differential privacy, 2021
Nicolas Papernot and Thomas Steinke · 2021
Closest in time.
Privacy-adaptive bert for natural language understanding
Chen Qu, Weize Kong, Liu Yang, Mingyang Zhang, Michael Bendersky, and Marc Najork · 2021
Closest in time.
Finetuned language models are zero-shot learners
Jason Wei, Maarten Bosma, Vincent Y Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoc V Le · 2021
Closest in time.
On a utilitarian approach to privacy preserving text generation
Zekun Xu, Abhinav Aggarwal, Oluwaseyi Feyisetan, and Nathanael Teissier · 2021
Closest in time.
Opacus: User-friendly differential privacy library in pytorch, 2021
Ashkan Yousefpour, Igor Shilov, Alexandre Sablayrolles, Davide Testuggine, Karthik Prasad, Mani Malek, John Nguyen, Sayan Gosh, Akash Bharadwaj, Jessica Zhao, Graham Cormode, and Ilya Mironov · 2021
Closest in time.
Wide network learning with differential privacy
Huanyu Zhang, Ilya Mironov, and Meisam Hejazinia · 2021
Closest in time.