Fetching the paper…
Reading the bibliography…
We study the problem of differentially private (DP) fine-tuning of large pre-trained models -- a recent privacy-preserving approach suitable for solving downstream tasks with sensitive data.
Rényi differential privacy of the sampled gaussian mechanism
Mironov, I., Talwar, K., and Zhang, L · 1908
Earlier work this paper cites.
Acceleration of stochastic approximation by averaging
Polyak, B. T. and Juditsky, A. B · 1992
Earlier work this paper cites.
Bleu: a method for automatic evaluation of machine translation
Papineni, K., Roukos, S., Ward, T., and jing Zhu, W · 2002
Earlier work this paper cites.
ROUGE: A package for automatic evaluation of summaries
Lin, C.-Y · 2004
Earlier work this paper cites.
METEOR: An automatic metric for MT evaluation with improved correlation with human judgments
Banerjee, S. and Lavie, A · 2005
Earlier work this paper cites.
Calibrating noise to sensitivity in private data analysis
Dwork, C., McSherry, F., Nissim, K., and Smith, A · 2006
Earlier work this paper cites.
A survey on transfer learning
Pan, S. J. and Yang, Q · 2009
Earlier work this paper cites.
Recursive deep models for semantic compositionality over a sentiment treebank
Socher, R., Perelygin, A., Wu, J., Chuang, J., Manning, C. D., Ng, A., and Potts, C · 2013
Earlier work this paper cites.
Efficient per-example gradient computations
Goodfellow, I · 2015
Earlier work this paper cites.
Cider: Consensus-based image description evaluation
Vedantam, R., Lawrence Zitnick, C., and Parikh, D · 2015
Earlier work this paper cites.
Deep learning with differential privacy
Abadi, M., Chu, A., Goodfellow, I., McMahan, H. B., Mironov, I., Talwar, K., and Zhang, L · 2016
Earlier work this paper cites.
Training deep nets with sublinear memory cost
Chen, T., Xu, B., Zhang, C., and Guestrin, C · 2016
Earlier work this paper cites.
Deep residual learning for image recognition
He, K., Zhang, X., Ren, S., and Sun, J · 2016
Earlier work this paper cites.
Squad: 100,000+ questions for machine comprehension of text
Rajpurkar, P., Zhang, J., Lopyrev, K., and Liang, P · 2016
Earlier work this paper cites.
Accurate, large minibatch sgd: Training imagenet in 1 hour
Goyal, P., Dollár, P., Girshick, R., Noordhuis, P., Wesolowski, L., Kyrola, A., Tulloch, A., Jia, Y., and He, K · 2017
Earlier work this paper cites.
First quora dataset release: Question pairs, 2017
Iyer, S., Dandekar, N., and Csernai, K · 2017
Earlier work this paper cites.
Learning multiple visual domains with residual adapters
Rebuffi, S.-A., Bilen, H., and Vedaldi, A · 2017
Earlier work this paper cites.
Membership inference attacks against machine learning models
Shokri, R., Stronati, M., Song, C., and Shmatikov, V · 2017
Earlier work this paper cites.
Revisiting unreasonable effectiveness of data in deep learning era
Sun, C., Shrivastava, A., Singh, S., and Gupta, A · 2017
Earlier work this paper cites.
Progressive growing of gans for improved quality, stability, and variation
Karras, T., Aila, T., Laine, S., and Lehtinen, J · 2018
Earlier work this paper cites.
The 2017 nist language recognition evaluation
Sadjadi, S. O., Kheyrkhah, T., Tong, A., Greenberg, C. S., Reynolds, D. A., Singer, E., Mason, L. P., Hernandez-Cordero, J., et al · 2018
Earlier work this paper cites.
A broad-coverage challenge corpus for sentence understanding through inference
Williams, A., Nangia, N., and Bowman, S · 2018
Earlier work this paper cites.
Evaluating the State-of-the-Art of End-to-End Natural Language Generation: The E2E NLG Challenge
Dusek, O., Novikova, J., and Rieser, V · 2019
Cited alongside, same era.
Parameter-efficient transfer learning for nlp
Houlsby, N., Giurgiu, A., Jastrzebski, S., Morrone, B., De Laroussilhe, Q., Gesmundo, A., Attariyan, M., and Gelly, S · 2019
Cited alongside, same era.
A style-based generator architecture for generative adversarial networks
Karras, T., Laine, S., and Aila, T · 2019
Cited alongside, same era.
Bert: Pre-training of deep bidirectional transformers for language understanding
Kenton, J. D. M.-W. C. and Toutanova, L. K · 2019
Cited alongside, same era.
Do better imagenet models transfer better?
Kornblith, S., Shlens, J., and Le, Q. V · 2019
Cited alongside, same era.
Numerical composition of differential privacy
Gopi, S., Lee, Y. T., and Wutschitz, L · 2021
Later among the works it cites.
Towards a unified view of parameter-efficient transfer learning
He, J., Zhou, C., Ma, X., Berg-Kirkpatrick, T., and Neubig, G · 2021
Later among the works it cites.
Lora: Low-rank adaptation of large language models
Hu, E. J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., and Chen, W · 2021
Later among the works it cites.
The power of scale for parameter-efficient prompt tuning
Lester, B., Al-Rfou, R., and Constant, N · 2021
Later among the works it cites.
Datasets: A community library for natural language processing
Lhoest, Q., Villanova del Moral, A., Jernite, Y., Thakur, A., von Platen, P., Patil, S., Chaumond, J., Drame, M., Plu, J., Tunstall, L., Davison, J., Sasko, M., Chhablani, G., Malik, B., Brandeis, S., Le Scao, T., Sanh, V., Xu, C., Patry, N., McMillan-Major, A., Schmid, P., Gugger, S., Delangue, C., Matussière, T., Debut, L., Bekman, S., Cistac, P., Goehringer, T., Mustar, V., Lagunas, F., Rush, A., and Wolf, T · 2021
Later among the works it cites.
alphaXiv searches the wider corpus for related work and actual follow-ups.
alphaXiv is searching for related work…
Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., and Stoyanov, V · 2019
Cited alongside, same era.
Micro-batch training with batch-channel normalization and weight standardization
Qiao, S., Wang, H., Liu, C., Shen, W., and Yuille, A · 2019
Cited alongside, same era.
Subsampled rényi differential privacy and analytical moments accountant
Wang, Y.-X., Balle, B., and Kasiviswanathan, S. P · 2019
Cited alongside, same era.
Longformer: The long-document transformer
Beltagy, I., Peters, M. E., and Cohan, A · 2020
Cited alongside, same era.
Language models are few-shot learners
Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al · 2020
Cited alongside, same era.
Deep learning with gaussian differential privacy
Bu, Z., Dong, J., Long, Q., and Su, W. J · 2020
Cited alongside, same era.
Tinytl: Reduce memory, not parameters for efficient on-device learning
Cai, H., Gan, C., Zhu, L., and Han, S · 2020
Cited alongside, same era.
Large language models can be strong differentially private learners
Li, X., Tramer, F., Liang, P., and Hashimoto, T · 2021
Later among the works it cites.
Prefix-tuning: Optimizing continuous prompts for generation
Li, X. L. and Liang, P · 2021
Later among the works it cites.
Liu, X., Zheng, Y., Du, Z., Ding, M., Qian, Y., Yang, Z., and Tang, J · 2021
Later among the works it cites.
Compacter: Efficient low-rank hypercomplex adapter layers
Mahabadi, R. K., Henderson, J., and Ruder, S · 2021
Later among the works it cites.
Adapterfusion: Non-destructive task composition for transfer learning
Pfeiffer, J., Kamath, A., Rücklé, A., Cho, K., and Gurevych, I · 2021
Later among the works it cites.
Adapterdrop: On the efficiency of adapters in transformers
Rücklé, A., Geigle, G., Glockner, M., Beck, T., Pfeiffer, J., Reimers, N., and Gurevych, I · 2021
Later among the works it cites.
Enabling fast differentially private sgd via just-in-time compilation and vectorization
Subramani, P., Vadivelu, N., and Kamath, G · 2021
Later among the works it cites.
Opacus: User-friendly differential privacy library in PyTorch
Yousefpour, A., Shilov, I., Sablayrolles, A., Testuggine, D., Prasad, K., Malek, M., Nguyen, J., Ghosh, S., Bharadwaj, A., Zhao, J., Cormode, G., and Mironov, I · 2021
Later among the works it cites.
Unlocking high-accuracy differentially private image classification through scale
De, S., Berrada, L., Hayes, J., Smith, S. L., and Balle, B · 2022
Closest in time.
Reconstructing training data from trained neural networks
Haim, N., Vardi, G., Yehudai, G., Shamir, O., and Irani, M · 2022
Closest in time.
Toward training at imagenet scale with differential privacy
Kurakin, A., Chien, S., Song, S., Geambasu, R., Terzis, A., and Thakurta, A · 2022
Closest in time.
Large scale transfer learning for differentially private image classification
Mehta, H., Thakurta, A., Kurakin, A., and Cutkosky, A · 2022
Closest in time.
Normalized/clipped sgd with perturbation for differentially private non-convex optimization
Yang, X., Zhang, H., Chen, W., and Liu, T.-Y · 2022
Closest in time.
Bitfit: Simple parameter-efficient fine-tuning for transformer-based masked language-models
Zaken, E. B., Goldberg, Y., and Ravfogel, S · 2022
Closest in time.
Differentially private optimization on large model at small cost
Bu, Z., Wang, Y.-X., Zha, S., and Karypis, G · 2023
Closest in time.
Palm: Scaling language modeling with pathways
Chowdhery, A., Narang, S., Devlin, J., Bosma, M., Mishra, G., Roberts, A., Barham, P., Chung, H. W., Sutton, C., Gehrmann, S., et al · 2023
Closest in time.