Fetching the paper…
Reading the bibliography…
The widespread practice of fine-tuning large language models (LLMs) on domain-specific data faces two major challenges in memory and privacy.
RoBERTa: A robustly optimized BERT pretraining approach
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov · 1907
Earlier work this paper cites.
A note on a method for generating points uniformly on n n -dimensional spheres
Mervin E Muller · 1959
Earlier work this paper cites.
A topological property of real analytic subsets
Stanislaw Łojasiewicz · 1963
Earlier work this paper cites.
Gradient methods for the minimisation of functionals
Boris T Polyak · 1963
Earlier work this paper cites.
Choosing a point from the surface of a sphere
George Marsaglia · 1972
Earlier work this paper cites.
Mathematical methods of statistics , volume 43
Harald Cramér · 1999
Earlier work this paper cites.
Building a question answering test collection
Ellen M Voorhees and Dawn M Tice · 2000
Earlier work this paper cites.
Finite-time analysis of the multiarmed bandit problem
Peter Auer, Nicolo Cesa-Bianchi, and Paul Fischer · 2002
Earlier work this paper cites.
BLEU: A method for automatic evaluation of machine translation
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu · 2002
Earlier work this paper cites.
Introductory lectures on convex optimization: A basic course , volume 87
Yurii Nesterov · 2003
Earlier work this paper cites.
Handbook of Beta distribution and its applications
Arjun K Gupta and Saralees Nadarajah · 2004
Earlier work this paper cites.
The PASCAL recognising textual entailment challenge, 2005
Ido Dagan, Oren Glickman, and Bernardo Magnini · 2005
Earlier work this paper cites.
Online convex optimization in the bandit setting: Gradient descent without a gradient
Abraham D Flaxman, Adam Tauman Kalai, and H Brendan McMahan · 2005
Earlier work this paper cites.
Calibrating noise to sensitivity in private data analysis
Cynthia Dwork, Frank McSherry, Kobbi Nissim, and Adam Smith · 2006
Earlier work this paper cites.
The second PASCAL recognising textual entailment challenge, 2006
R Bar Haim, Ido Dagan, Bill Dolan, Lisa Ferro, Danilo Giampiccolo, Bernardo Magnini, and Idan Szpektor · 2006
Earlier work this paper cites.
The third PASCAL recognizing textual entailment challenge, 2007
Danilo Giampiccolo, Bernardo Magnini, Ido Dagan, and William B Dolan · 2007
Earlier work this paper cites.
The fifth PASCAL recognizing textual entailment challenge, 2009
Luisa Bentivogli, Peter Clark, Ido Dagan, and Danilo Giampiccolo · 2009
Earlier work this paper cites.
Differentially private empirical risk minimization
Kamalika Chaudhuri, Claire Monteleoni, and Anand D Sarwate · 2011
Earlier work this paper cites.
Randomized smoothing for stochastic optimization
John C Duchi, Peter L Bartlett, and Martin J Wainwright · 2012
Earlier work this paper cites.
Finite sample convergence rates of zero-order stochastic optimization methods
Andre Wibisono, Martin J Wainwright, Michael Jordan, and John C Duchi · 2012
Earlier work this paper cites.
On stochastic gradient and subgradient methods with adaptive steplength sequences
Farzad Yousefian, Angelia Nedić, and Uday V Shanbhag · 2012
Earlier work this paper cites.
Stochastic first-and zeroth-order methods for nonconvex stochastic programming
Saeed Ghadimi and Guanghui Lan · 2013
Earlier work this paper cites.
(Nearly) optimal algorithms for private online learning in full-information and bandit settings
Abhradeep Guha Thakurta and Adam Smith · 2013
Earlier work this paper cites.
Privacy via the Johnson-Lindenstrauss transform
Krishnaram Kenthapadi, Aleksandra Korolova, Ilya Mironov, and Nina Mishra · 2013
Earlier work this paper cites.
Recursive deep models for semantic compositionality over a sentiment treebank
Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D Manning, Andrew Y Ng, and Christopher Potts · 2013
Earlier work this paper cites.
Stochastic gradient descent with differentially private updates
Shuang Song, Kamalika Chaudhuri, and Anand D Sarwate · 2013
Earlier work this paper cites.
Private empirical risk minimization: Efficient algorithms and tight error bounds
Raef Bassily, Adam Smith, and Abhradeep Thakurta · 2014
Earlier work this paper cites.
The algorithmic foundations of differential privacy
Cynthia Dwork, Aaron Roth, et al · 2014
Earlier work this paper cites.
(Near) dimension independent risk bounds for differentially private learning
Prateek Jain and Abhradeep Guha Thakurta · 2014
Earlier work this paper cites.
A large annotated corpus for learning natural language inference
Samuel R Bowman, Gabor Angeli, Christopher Potts, and Christopher D Manning · 2015
Earlier work this paper cites.
Optimal rates for zero-order convex optimization: The power of two function evaluations
John C Duchi, Michael I Jordan, Martin J Wainwright, and Andre Wibisono · 2015
Earlier work this paper cites.
Black-box optimization of noisy functions with unknown smoothness
Jean-Bastien Grill, Michal Valko, and Rémi Munos · 2015
Earlier work this paper cites.
The composition theorem for differential privacy
Peter Kairouz, Sewoong Oh, and Pramod Viswanath · 2015
Earlier work this paper cites.
Deep learning with differential privacy
Martin Abadi, Andy Chu, Ian Goodfellow, H Brendan McMahan, Ilya Mironov, Kunal Talwar, and Li Zhang · 2016
Earlier work this paper cites.
Linear convergence of gradient and proximal-gradient methods under the Polyak-Łojasiewicz condition
Hamed Karimi, Julie Nutini, and Mark Schmidt · 2016
Earlier work this paper cites.
A comprehensive linear speedup analysis for asynchronous stochastic parallel optimization from zeroth-order to first-order
Xiangru Lian, Huan Zhang, Cho-Jui Hsieh, Yijun Huang, and Ji Liu · 2016
Earlier work this paper cites.
SQuAD: 100,000+ questions for machine comprehension of text
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang · 2016
Earlier work this paper cites.
Algorithms for differentially private multi-armed bandits
Aristide Tossou and Christos Dimitrakakis · 2016
Earlier work this paper cites.
ZOO: Zeroth order optimization based black-box attacks to deep neural networks without training substitute models
Pin-Yu Chen, Huan Zhang, Yash Sharma, Jinfeng Yi, and Cho-Jui Hsieh · 2017
Earlier work this paper cites.
Random gradient-free minimization of convex functions
Yurii Nesterov and Vladimir Spokoiny · 2017
Earlier work this paper cites.
Empirical analysis of the Hessian of over-parametrized neural networks
Levent Sagun, Utku Evci, V Ugur Guney, Yann Dauphin, and Leon Bottou · 2017
Earlier work this paper cites.
Evolution strategies as a scalable alternative to reinforcement learning
Tim Salimans, Jonathan Ho, Xi Chen, Szymon Sidor, and Ilya Sutskever · 2017
Earlier work this paper cites.
An optimal algorithm for bandit and zero-order convex optimization with two-point feedback
Ohad Shamir · 2017
Earlier work this paper cites.
Membership inference attacks against machine learning models
Reza Shokri, Marco Stronati, Congzheng Song, and Vitaly Shmatikov · 2017
Cited alongside, same era.
The EU general data protection regulation (GDPR)
Paul Voigt and Axel Von dem Bussche · 2017
Cited alongside, same era.
Differentially private empirical risk minimization revisited: Faster and more general
Di Wang, Minwei Ye, and Jinhui Xu · 2017
Cited alongside, same era.
Bolt-on differential privacy for scalable stochastic gradient descent-based analytics
Xi Wu, Fengan Li, Arun Kumar, Kamalika Chaudhuri, Somesh Jha, and Jeffrey Naughton · 2017
Cited alongside, same era.
Efficient private ERM for smooth objectives
Jiaqi Zhang, Kai Zheng, Wenlong Mou, and Liwei Wang · 2017
Cited alongside, same era.
Zeroth-order (non)-convex stochastic optimization via conditional gradient and gradient updates
Private non-smooth ERM and SCO in subquadratic steps
Janardhan Kulkarni, Yin Tat Lee, and Daogao Liu · 2021
Later among the works it cites.
Prefix-tuning: Optimizing continuous prompts for generation
Xiang Lisa Li and Percy Liang · 2021
Later among the works it cites.
Differential privacy over Riemannian manifolds
Matthew Reimherr, Karthik Bharath, and Carlos Soto · 2021
Later among the works it cites.
Evading the curse of dimensionality in unconstrained private GLMs
Shuang Song, Thomas Steinke, Om Thakkar, and Abhradeep Thakurta · 2021
Later among the works it cites.
Bypassing the ambient dimension: Private SGD with gradient subspace identification
Yingxue Zhou, Steven Wu, and Arindam Banerjee · 2021
Later among the works it cites.
alphaXiv searches the wider corpus for related work and actual follow-ups.
alphaXiv is searching for related work…
Krishnakumar Balasubramanian and Saeed Ghadimi · 2018
Cited alongside, same era.
Structured evolution with compact architectures for scalable policy optimization
Krzysztof Choromanski, Mark Rowland, Vikas Sindhwani, Richard Turner, and Adrian Weller · 2018
Cited alongside, same era.
SPIDER: Near-optimal non-convex optimization via stochastic path-integrated differential estimator
Cong Fang, Chris Junchi Li, Zhouchen Lin, and Tong Zhang · 2018
Cited alongside, same era.
Gradient descent happens in a tiny subspace
Guy Gur-Ari, Daniel A Roberts, and Ethan Dyer · 2018
Cited alongside, same era.
Measuring the intrinsic dimension of objective landscapes
Chunyuan Li, Heerad Farkhoor, Rosanne Liu, and Jason Yosinski · 2018
Cited alongside, same era.
Zeroth-order stochastic variance reduction for nonconvex optimization
Sijia Liu, Bhavya Kailkhura, Pin-Yu Chen, Paishun Ting, Shiyu Chang, and Lisa Amini · 2018
Cited alongside, same era.
Decoupled weight decay regularization
Ilya Loshchilov and Frank Hutter · 2018
Cited alongside, same era.
Atılım Güneş Baydin, Barak A Pearlmutter, Don Syme, Frank Wood, and Philip Torr · 2022
Later among the works it cites.
Zeroth-order regularized optimization (ZORO): Approximately sparse gradients and adaptive sampling
HanQin Cai, Daniel Mckenzie, Wotao Yin, and Zhenliang Zhang · 2022
Later among the works it cites.
Communication-efficient stochastic zeroth-order optimization for federated learning
Wenzhi Fang, Ziyi Yu, Yuning Jiang, Yuanming Shi, Colin N Jones, and Yong Zhou · 2022
Later among the works it cites.
The forward-forward algorithm: Some preliminary investigations
Geoffrey Hinton · 2022
Later among the works it cites.
LoRA: Low-rank adaptation of large language models
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen · 2022
Later among the works it cites.
Gradient-free methods for deterministic and stochastic nonsmooth nonconvex optimization
Tianyi Lin, Zeyu Zheng, and Michael Jordan · 2022
Later among the works it cites.
DP-PCA: Statistically optimal and differentially private PCA
Xiyang Liu, Weihao Kong, Prateek Jain, and Sewoong Oh · 2022
Later among the works it cites.
Dimension independent generalization of DP-SGD for overparameterized smooth convex optimization
Yi-An Ma, Teodor Vanislavov Marinov, and Tong Zhang · 2022
Later among the works it cites.
Memorization in NLP fine-tuning methods
Fatemehsadat Mireshghallah, Archit Uniyal, Tianhao Wang, David Evans, and Taylor Berg-Kirkpatrick · 2022
Later among the works it cites.
Black-box generalization: Stability of zeroth-order learning
Konstantinos Nikolakakis, Farzin Haddadpour, Dionysis Kalogerias, and Amin Karbasi · 2022
Later among the works it cites.
Training language models to follow instructions with human feedback
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al · 2022
Later among the works it cites.
Learning by directional gradient descent
David Silver, Anirudh Goyal, Ivo Danihelka, Matteo Hessel, and Hado van Hasselt · 2022
Later among the works it cites.
Momentum aggregation for private non-convex ERM
Hoang Tran and Ashok Cutkosky · 2022
Later among the works it cites.
Zeroth-order algorithms for nonconvex–strongly-concave minimax problems with improved complexities
Zhongruo Wang, Krishnakumar Balasubramanian, Shiqian Ma, and Meisam Razaviyayn · 2022
Later among the works it cites.
Normalized/Clipped SGD with perturbation for differentially private non-convex optimization
Xiaodong Yang, Huishuai Zhang, Wei Chen, and Tie-Yan Liu · 2022
Later among the works it cites.
Differentially private fine-tuning of language models
Da Yu, Saurabh Naik, Arturs Backurs, Sivakanth Gopi, Huseyin A Inan, Gautam Kamath, Janardhan Kulkarni, Yin Tat Lee, Andre Manoel, Lukas Wutschitz, Sergey Yekhanin, and Huishuai Zhang · 2022
Later among the works it cites.
Faster rates of convergence to stationary points in differentially private optimization
Raman Arora, Raef Bassily, Tomás González, Cristóbal A Guzmán, Michael Menart, and Enayat Ullah · 2023
Closest in time.
DP-Forward: Fine-tuning and inference on language models with differential privacy in forward pass
Minxin Du, Xiang Yue, Sherman SM Chow, Tianhao Wang, Chenyu Huang, and Huan Sun · 2023
Closest in time.
Improved convergence of differential private SGD with gradient clipping
Huang Fang, Xiaoyun Li, Chenglin Fan, and Ping Li · 2023
Closest in time.
Exploring the limits of differentially private deep learning with group-wise clipping
Jiyan He, Xuechen Li, Da Yu, Huishuai Zhang, Janardhan Kulkarni, Yin Tat Lee, Arturs Backurs, Nenghai Yu, and Jiang Bian · 2023
Closest in time.
PromptBoosting: Black-box text classification with ten forward passes
Bairu Hou, Joe O’connor, Jacob Andreas, Shiyu Chang, and Yang Zhang · 2023
Closest in time.
Revisiting gradient clipping: Stochastic bias and tight convergence guarantees
Anastasia Koloskova, Hadrien Hendrikx, and Sebastian U Stich · 2023
Closest in time.
Optimal differentially private learning with public data
Andrew Lowy, Zeman Li, Tianjian Huang, and Meisam Razaviyayn · 2023
Closest in time.
Analyzing leakage of personally identifiable information in language models
Nils Lukas, Ahmed Salem, Robert Sim, Shruti Tople, Lukas Wutschitz, and Santiago Zanella-Béguelin · 2023
Closest in time.
Fine-tuning language models with just forward passes
Sadhika Malladi, Tianyu Gao, Eshaan Nichani, Alex Damian, Jason D Lee, Danqi Chen, and Sanjeev Arora · 2023
Closest in time.
Membership inference attacks against language models via neighbourhood comparison
Justus Mattern, Fatemehsadat Mireshghallah, Zhijing Jin, Bernhard Schölkopf, Mrinmaya Sachan, and Taylor Berg-Kirkpatrick · 2023
Closest in time.
GPT-4 Technical Report, 2023
OpenAI · 2023
Closest in time.
HyperTuning: Toward adapting large language models without back-propagation
Jason Phang, Yi Mao, Pengcheng He, and Weizhu Chen · 2023
Closest in time.
Share your representation only: Guaranteed improvement of the privacy-utility tradeoff in federated learning
Zebang Shen, Jiayuan Ye, Anmin Kang, Hamed Hassani, and Reza Shokri · 2023
Closest in time.
Federated fine-tuning of billion-sized language models across mobile devices
Mengwei Xu, Yaozong Wu, Dongqi Cai, Xiang Li, and Shangguang Wang · 2023
Closest in time.
Zeroth-order optimization with weak dimension dependency
Pengyun Yue, Long Yang, Cong Fang, and Zhouchen Lin · 2023
Closest in time.
Eric Zelikman, Qian Huang, Percy Liang, Nick Haber, and Noah D Goodman · 2023
Closest in time.
Exploring memorization in fine-tuned language models
Shenglai Zeng, Yaxin Li, Jie Ren, Yiding Liu, Han Xu, Pengfei He, Yue Xing, Shuaiqiang Wang, Jiliang Tang, and Dawei Yin · 2023
Closest in time.
DPZero: Dimension-independent and differentially private zeroth-order optimization
Liang Zhang, Kiran K Thekumparampil, Sewoong Oh, and Niao He · 2023
Closest in time.
Differentially private Riemannian optimization
Andi Han, Bamdev Mishra, Pratik Jawanpuria, and Junbin Gao · 2024
Closest in time.
DP-OPT: Make large language model your privacy-preserving prompt engineer
Junyuan Hong, Jiachen T. Wang, Chenhui Zhang, Zhangheng LI, Bo Li, and Zhangyang Wang · 2024
Closest in time.
Differentially private zeroth-order methods for scalable large language model finetuning
Zhihao Liu, Jian Lou, Wenjie Bao, Yuke Hu, Bo Li, Zhan Qin, and Kui Ren · 2024
Closest in time.