Fetching the paper…
Reading the bibliography…
Differentially Private methods for training Deep Neural Networks (DNNs) have progressed recently, in particular with the use of massive batches and aggregated data augmentations for a large number of training steps.
A simple framework for contrastive learning of visual representations
Chen, T., Kornblith, S., Norouzi, M., and Hinton, G. E · 2002
Earlier work this paper cites.
Language models are few-shot learners, 2020
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., Agarwal, S., Herbert-Voss, A., Krueger, G., Henighan, T., Child, R., Ramesh, A., Ziegler, D. M., Wu, J., Winter, C., Hesse, C., Chen, M., Sigler, E., Litwin, M., Gray, S., Chess, B., Clark, J., Berner, C., McCandlish, S., Radford, A., Sutskever, I., and Amodei, D · 2005
Earlier work this paper cites.
Calibrating noise to sensitivity in private data analysis
Dwork, C., McSherry, F., Nissim, K., and Smith, A · 2006
Earlier work this paper cites.
On the generalization benefit of noise in stochastic gradient descent
Smith, S. L., Elsen, E., and De, S · 2006
Earlier work this paper cites.
Imagenet: A large-scale hierarchical image database
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., and Fei-Fei, L · 2009
Earlier work this paper cites.
Learning multiple layers of features from tiny images
Krizhevsky, A., Hinton, G., et al · 2009
Earlier work this paper cites.
An image is worth 16x16 words: Transformers for image recognition at scale, 2020
Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., Uszkoreit, J., and Houlsby, N · 2010
Earlier work this paper cites.
Training data-efficient image transformers & distillation through attention
Touvron, H., Cord, M., Douze, M., Massa, F., Sablayrolles, A., and Jégou, H · 2012
Earlier work this paper cites.
Private empirical risk minimization: Efficient algorithms and tight error bounds
Bassily, R., Smith, A., and Thakurta, A · 2014
Earlier work this paper cites.
Imagenet large scale visual recognition challenge, 2014
Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., Berg, A. C., and Fei-Fei, L · 2014
Earlier work this paper cites.
Batch normalization: Accelerating deep network training by reducing internal covariate shift
Ioffe, S. and Szegedy, C · 2015
Earlier work this paper cites.
Deep learning with differential privacy
Abadi, M., Chu, A., Goodfellow, I., McMahan, H. B., Mironov, I., Talwar, K., and Zhang, L · 2016
Earlier work this paper cites.
Deep speech 2: End-to-end speech recognition in english and mandarin
Amodei, D., Ananthanarayanan, S., Anubhai, R., Bai, J., Battenberg, E., Case, C., Casper, J., Catanzaro, B., Cheng, Q., Chen, G., et al · 2016
Earlier work this paper cites.
Ba, J. L., Kiros, J. R., and Hinton, G. E · 2016
Earlier work this paper cites.
Concentrated differential privacy: Simplifications, extensions, and lower bounds
Bun, M. and Steinke, T · 2016
Earlier work this paper cites.
Concentrated differential privacy
Dwork, C. and Rothblum, G. N · 2016
Earlier work this paper cites.
Deep residual learning for image recognition
He, K., Zhang, X., Ren, S., and Sun, J · 2016
Earlier work this paper cites.
On large-batch training for deep learning: Generalization gap and sharp minima
Keskar, N. S., Mudigere, D., Nocedal, J., Smelyanskiy, M., and Tang, P. T. P · 2016
Cited alongside, same era.
Zagoruyko, S. and Komodakis, N · 2016
Cited alongside, same era.
Accurate, large minibatch sgd: Training imagenet in 1 hour
Goyal, P., Dollár, P., Girshick, R., Noordhuis, P., Wesolowski, L., Kyrola, A., Tulloch, A., Jia, Y., and He, K · 2017
Cited alongside, same era.
Rényi differential privacy
Mironov, I · 2017
Cited alongside, same era.
Differentially private empirical risk minimization revisited: Faster and more general
Wang, D., Ye, M., and Xu, J · 2017
Cited alongside, same era.
Subsampled rényi differential privacy and analytical moments accountant
Wang, Y.-X., Balle, B., and Kasiviswanathan, S. P · 2019
Later among the works it cites.
Pytorch image models
Wightman, R · 2019
Later among the works it cites.
Hypothesis testing interpretations and Rényi differential privacy
Balle, B., Barthe, G., Gaboardi, M., Hsu, J., and Sato, T · 2020
Later among the works it cites.
Deep learning with gaussian differential privacy
Bu, Z., Dong, J., Long, Q., and Su, W. J · 2020
Later among the works it cites.
Differentially private learning needs better features (or much more data)
Tramer, F. and Boneh, D · 2020
Later among the works it cites.
Large-scale differentially private bert, 2021
Anil, R., Ghazi, B., Gupta, V., Kumar, R., and Manurangsi, P · 2021
alphaXiv searches the wider corpus for related work and actual follow-ups.
alphaXiv is searching for related work…
Composable and versatile privacy via truncated cdp
Bun, M., Dwork, C., Rothblum, G. N., and Steinke, T · 2018
Cited alongside, same era.
Bert: Pre-training of deep bidirectional transformers for language understanding
Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K · 2018
Cited alongside, same era.
Privacy amplification by iteration
Feldman, V., Mironov, I., Talwar, K., and Thakurta, A · 2018
Cited alongside, same era.
Revisiting small batch training for deep neural networks
Masters, D. and Luschi, C · 2018
Cited alongside, same era.
Wu, Y. and He, K · 2018
Cited alongside, same era.
The secret sharer: Evaluating and testing unintended memorization in neural networks
Carlini, N., Liu, C., Erlingsson, Ú., Kos, J., and Song, D · 2019
Cited alongside, same era.
Sgd: General analysis and improved rates
Gower, R. M., Loizou, N., Qian, X., Sailanbayev, A., Shulgin, E., and Richtárik, P · 2019
Cited alongside, same era.
Later among the works it cites.
Extracting training data from large language models
Carlini, N., Tramer, F., Wallace, E., Jagielski, M., Herbert-Voss, A., Lee, K., Roberts, A., Brown, T., Song, D., Erlingsson, U., et al · 2021
Later among the works it cites.
Numerical composition of differential privacy
Gopi, S., Lee, Y. T., and Wutschitz, L · 2021
Later among the works it cites.
Large language models can be strong differentially private learners, 2021
Li, X., Tramèr, F., Liang, P., and Hashimoto, T · 2021
Later among the works it cites.
Implicit bias of sgd for diagonal linear networks: a provable benefit of stochasticity
Pesme, S., Pillaud-Vivien, L., and Flammarion, N · 2021
Later among the works it cites.
Scaling language models: Methods, analysis & insights from training gopher, 2021
Rae, J. W., Borgeaud, S., Cai, T., Millican, K., Hoffmann, J., Song, F., Aslanides, J., Henderson, S., Ring, R., Young, S., Rutherford, E., Hennigan, T., Menick, J., Cassirer, A., Powell, R., Driessche, G. v. d., Hendricks, L. A., Rauh, M., Huang, P.-S., Glaese, A., Welbl, J., Dathathri, S., Huang, S., Uesato, J., Mellor, J., Higgins, I., Creswell, A., McAleese, N., Wu, A., Elsen, E., Jayakumar, S., Buchatskaya, E., Budden, D., Sutherland, E., Simonyan, K., Paganini, M., Sifre, L., Martens, L., Li, X. L., Kuncoro, A., Nematzadeh, A., Gribovskaya, E., Donato, D., Lazaridou, A., Mensch, A., Lespiau, J.-B., Tsimpoukelli, M., Grigorev, N., Fritz, D., Sottiaux, T., Pajarskas, M., Pohlen, T., Gong, Z., Toyama, D., d’Autume, C. d. M., Li, Y., Terzi, T., Mikulik, V., Babuschkin, I., Clark, A., Casas, D. d. L., Guy, A., Jones, C., Bradbury, J., Johnson, M., Hechtman, B., Weidinger, L., Gabriel, I., Isaac, W., Lockhart, E., Osindero, S., Rimell, L., Dyer, C., Vinyals, O., Ayoub, K., Stanway, J., Bennett, L., Hassabis, D., Kavukcuoglu, K., and Irving, G · 2021
Later among the works it cites.
Opacus: User-friendly differential privacy library in PyTorch, 2021
Yousefpour, A., Shilov, I., Sablayrolles, A., Testuggine, D., Prasad, K., Malek, M., Nguyen, J., Ghosh, S., Bharadwaj, A., Zhao, J., Cormode, G., and Mironov, I · 2021
Later among the works it cites.
Unlocking high-accuracy differentially private image classification through scale, 2022
De, S., Berrada, L., Hayes, J., Smith, S. L., and Balle, B · 2022
Closest in time.
Toward training at imagenet scale with differential privacy, 2022
Kurakin, A., Song, S., Chien, S., Geambasu, R., Terzis, A., and Thakurta, A · 2022
Closest in time.
Hierarchical text-conditional image generation with clip latents, 2022
Ramesh, A., Dhariwal, P., Nichol, A., Chu, C., and Chen, M · 2022
Closest in time.
Differentially private learning needs hidden state (or much faster convergence)
Ye, J. and Shokri, R · 2022
Closest in time.