Differentially Private Zeroth-Order Methods for Scalable Large Language Model Finetuning

J. C. Spall, “Multivariate stochastic approximation using a simultaneous perturbation gradient approximation,” IEEE transactions on automatic control , vol. 37, no. 3, pp. 332–341, 1992

1992

Earlier work this paper cites.

E. M. Voorhees and D. M. Tice, “Building a question answering test collection,” in Proceedings of the 23rd annual international ACM SIGIR conference on Research and development in information retrieval , 2000, pp. 200–207

2000

Earlier work this paper cites.

C.-Y. Lin, “Rouge: A package for automatic evaluation of summaries,” in Text summarization branches out , 2004, pp. 74–81

2004

Earlier work this paper cites.

P. Koehn, “Europarl: A parallel corpus for statistical machine translation,” in Proceedings of machine translation summit x: papers , 2005, pp. 79–86

2005

Earlier work this paper cites.

C. Dwork, “Differential privacy,” in International colloquium on automata, languages, and programming . Springer, 2006, pp. 1–12

2006

Earlier work this paper cites.

C. Dwork, F. McSherry, K. Nissim, and A. Smith, “Calibrating noise to sensitivity in private data analysis,” in Theory of cryptography conference . Springer, 2006, pp. 265–284

2006

Earlier work this paper cites.

S. Ghadimi and G. Lan, “Stochastic first-and zeroth-order methods for nonconvex stochastic programming,” SIAM Journal on Optimization , vol. 23, no. 4, pp. 2341–2368, 2013

2013

Earlier work this paper cites.

R. Socher, A. Perelygin, J. Wu, J. Chuang, C. D. Manning, A. Y. Ng, and C. Potts, “Recursive deep models for semantic compositionality over a sentiment treebank,” in Proceedings of the 2013 conference on empirical methods in natural language processing , 2013, pp. 1631–1642

2013

Earlier work this paper cites.

M. Bun, J. Ullman, and S. Vadhan, “Fingerprinting codes and the price of approximate differential privacy,” in Proceedings of the forty-sixth annual ACM symposium on Theory of computing , 2014, pp. 1–10

2014

Earlier work this paper cites.

S. R. Bowman, G. Angeli, C. Potts, and C. D. Manning, “A large annotated corpus for learning natural language inference,” arXiv preprint arXiv:1508.05326 , 2015

Original

2015

Earlier work this paper cites.

M. Abadi, A. Chu, I. Goodfellow, H. B. McMahan, I. Mironov, K. Talwar, and L. Zhang, “Deep learning with differential privacy,” in Proceedings of the 2016 ACM SIGSAC conference on computer and communications security , 2016, pp. 308–318

2016

Earlier work this paper cites.

H. Karimi, J. Nutini, and M. Schmidt, “Linear convergence of gradient and proximal-gradient methods under the polyak-łojasiewicz condition,” in Machine Learning and Knowledge Discovery in Databases: European Conference, ECML PKDD 2016, Riva del Garda, Italy, September 19-23, 2016, Proceedings, Part I 16 . Springer, 2016, pp. 795–811

2016

Earlier work this paper cites.

P. Rajpurkar, “Squad: 100,000+ questions for machine comprehension of text,” arXiv preprint arXiv:1606.05250 , 2016

Original

2016

Earlier work this paper cites.

R. Nallapati, B. Zhou, C. Gulcehre, B. Xiang et al. , “Abstractive text summarization using sequence-to-sequence rnns and beyond,” arXiv preprint arXiv:1602.06023 , 2016

Original

2016

Earlier work this paper cites.

Y. Nesterov and V. Spokoiny, “Random gradient-free minimization of convex functions,” Foundations of Computational Mathematics , vol. 17, pp. 527–566, 2017

2017

Earlier work this paper cites.

A. Williams, N. Nangia, and S. R. Bowman, “A broad-coverage challenge corpus for sentence understanding through inference,” arXiv preprint arXiv:1704.05426 , 2017

Original

2017

Earlier work this paper cites.

A. Radford, K. Narasimhan, T. Salimans, I. Sutskever et al. , “Improving language understanding by generative pre-training,” 2018

2018

Earlier work this paper cites.

B. Balle, G. Barthe, and M. Gaboardi, “Privacy amplification by subsampling: Tight analyses via couplings and divergences,” Advances in neural information processing systems , vol. 31, 2018

2018

Earlier work this paper cites.

R. Bassily, M. Belkin, and S. Ma, “On exponential convergence of sgd in non-convex over-parametrized learning,” arXiv preprint arXiv:1811.02564 , 2018

Original

2018

Earlier work this paper cites.

D. Khashabi, S. Chaturvedi, M. Roth, S. Upadhyay, and D. Roth, “Looking beyond the surface: A challenge set for reading comprehension over multiple sentences,” in Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) , 2018, pp. 252–262

2018

Earlier work this paper cites.

S. Narayan, S. B. Cohen, and M. Lapata, “Don’t give me the details, just the summary! topic-aware convolutional neural networks for extreme summarization,” arXiv preprint arXiv:1808.08745 , 2018

Original

2018

Cited alongside, same era.

A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, I. Sutskever et al. , “Language models are unsupervised multitask learners,” OpenAI blog , vol. 1, no. 8, p. 9, 2019

2019

Cited alongside, same era.

D. M. Ziegler, N. Stiennon, J. Wu, T. B. Brown, A. Radford, D. Amodei, P. Christiano, and G. Irving, “Fine-tuning language models from human preferences,” arXiv preprint arXiv:1909.08593 , 2019

Original

2019

Cited alongside, same era.

C. Sun, X. Qiu, Y. Xu, and X. Huang, “How to fine-tune bert for text classification?” in Chinese Computational Linguistics: 18th China National Conference, CCL 2019, Kunming, China, October 18–20, 2019, Proceedings 18 . Springer, 2019, pp. 194–206

2019

Cited alongside, same era.

R. Bassily, C. Guzmán, and M. Menart, “Differentially private stochastic optimization: New results in convex and non-convex settings,” Advances in Neural Information Processing Systems , vol. 34, pp. 9317–9329, 2021

2021

Later among the works it cites.

S. Zhang, S. Roller, N. Goyal, M. Artetxe, M. Chen, S. Chen, C. Dewan, M. Diab, X. Li, X. V. Lin et al. , “Opt: Open pre-trained transformer language models,” arXiv preprint arXiv:2205.01068 , 2022

Original

2022

Later among the works it cites.

J. He, X. Li, D. Yu, H. Zhang, J. Kulkarni, Y. T. Lee, A. Backurs, N. Yu, and J. Bian, “Exploring the limits of differentially private deep learning with group-wise clipping,” arXiv preprint arXiv:2212.01539 , 2022

Original

2022

Later among the works it cites.

Z. Bu, Y.-X. Wang, S. Zha, and G. Karypis, “Automatic clipping: Differentially private deep learning made easier and stronger,” arXiv preprint arXiv:2206.07136 , 2022

Original

2022

Later among the works it cites.