2023

Gotcha! This Model Uses My Code! Evaluating Membership Leakage Risks in Code Models

Yang, Zhou, Zhao, Zhipeng, Wang, Chenyu et al.

Understand

Given large-scale source code datasets available in open-source projects and advanced large language models, recent code models have been proposed to address a series of critical software engineering tasks, such as program repair and code completion.

The training data of the code models come from various sources, not only the publicly available source code, e.g., open-source projects on GitHub but also the private data such as the confidential source code from companies, which may contain sensitive information (for example, SSH keys and personal information).
As a result, the use of these code models may raise new privacy concerns.
In this paper, we focus on a critical yet not well-explored question on using code models: what is the risk of membership information leakage in code models? Membership information leakage refers to the risk that an attacker can infer whether a given data point is included in (i.e., a member of) the training data.

Built on

Y. Liu, M. Ott, N. Goyal, J. Du, M. Joshi, D. Chen, O. Levy, M. Lewis, L. Zettlemoyer, and V. Stoyanov, “Roberta: A robustly optimized BERT pretraining approach,” CoRR , vol. abs/1907.11692, 2019. [Online]. Available: http://arxiv.org/abs/1907.11692
Original
1907
Earlier work this paper cites.
F. Wilcoxon, “Individual comparisons by ranking methods,” Biometrics Bulletin , vol. 1, no. 6, pp. 80–83, 1945. [Online]. Available: http://www.jstor.org/stable/3001968
1945
Earlier work this paper cites.
R. A. Fisher, R. A. Fisher, S. Genetiker, R. A. Fisher, S. Genetician, G. Britain, R. A. Fisher, and S. Généticien, The design of experiments . Springer, 1966, vol. 21
1966
Earlier work this paper cites.
J. Cohen, Statistical Power Analysis for the Behavioral Sciences , 2nd ed. Hillsdale, NJ: Lawrence Erlbaum Associates, 1988
1988
Earlier work this paper cites.
L. St, S. Wold et al. , “Analysis of variance (anova),” Chemometrics and intelligent laboratory systems , vol. 6, no. 4, pp. 259–272, 1989
1989
Earlier work this paper cites.
L. E. Toothaker, Multiple comparison procedures . Sage, 1993, no. 89
1993
Earlier work this paper cites.
S. F. Chen and J. Goodman, “An empirical study of smoothing techniques for language modeling,” Computer Speech and Language , vol. 13, no. 4, pp. 359–394, 1999. [Online]. Available: https://www.sciencedirect.com/science/article/pii/S0885230899901286
1999
Earlier work this paper cites.
C.-Y. Lin and F. J. Och, “Automatic evaluation of machine translation quality using longest common subsequence and skip-bigram statistics,” in Proceedings of the 42nd Annual Meeting on Association for Computational Linguistics , ser. ACL ’04. USA: Association for Computational Linguistics, 2004, p. 605–es. [Online]. Available: https://doi.org/10.3115/1218955.1219032
2004
Earlier work this paper cites.
D. Montgomery, Design and Analysis of Experiments 7th Edition with Student Solutions Manual and Design Expert 7. 0. 3 Set . John Wiley & Sons Canada, Limited, 2009. [Online]. Available: https://books.google.com.sg/books?id=S2CQuAAACAAJ
2009
Earlier work this paper cites.
P. Sedgwick, “Multiple significance tests: the bonferroni correction,” Bmj , vol. 344, 2012
2012
Earlier work this paper cites.
M. Allamanis and C. Sutton, “Mining source code repositories at massive scale using language modeling,” in 2013 10th IEEE Working Conference on Mining Software Repositories (MSR 2013) . Los Alamitos, CA, USA: IEEE Computer Society, may 2013, pp. 207–216. [Online]. Available: https://doi.ieeecomputersociety.org/10.1109/MSR.2013.6624029
2013
Earlier work this paper cites.
S. Bengio, O. Vinyals, N. Jaitly, and N. Shazeer, “Scheduled sampling for sequence prediction with recurrent neural networks,” Advances in neural information processing systems , vol. 28, 2015
2015
Earlier work this paper cites.
I. Goodfellow, J. Shlens, and C. Szegedy, “Explaining and harnessing adversarial examples,” in International Conference on Learning Representations , 2015. [Online]. Available: http://arxiv.org/abs/1412.6572
Original
2015
Earlier work this paper cites.
V. Raychev, P. Bielik, and M. Vechev, “Probabilistic model for code with decision trees,” in Proceedings of the 2016 ACM SIGPLAN International Conference on Object-Oriented Programming, Systems, Languages, and Applications , ser. OOPSLA 2016. New York, NY, USA: Association for Computing Machinery, 2016, p. 731–747. [Online]. Available: https://doi.org/10.1145/2983990.2984041
2016
Earlier work this paper cites.
M. Abadi, A. Chu, I. Goodfellow, H. B. McMahan, I. Mironov, K. Talwar, and L. Zhang, “Deep learning with differential privacy,” in Proceedings of the 2016 ACM SIGSAC Conference on Computer and Communications Security , ser. CCS ’16. New York, NY, USA: Association for Computing Machinery, 2016, p. 308–318. [Online]. Available: https://doi.org/10.1145/2976749.2978318
2016
Earlier work this paper cites.
F. Tramèr, F. Zhang, A. Juels, M. K. Reiter, and T. Ristenpart, “Stealing machine learning models via prediction apis,” in Proceedings of the 25th USENIX Conference on Security Symposium , ser. SEC’16. USA: USENIX Association, 2016, p. 601–618
2016
Earlier work this paper cites.
M. Freitag and Y. Al-Onaizan, “Beam search strategies for neural machine translation,” in Proceedings of the First Workshop on Neural Machine Translation . Vancouver: Association for Computational Linguistics, Aug. 2017, pp. 56–60. [Online]. Available: https://aclanthology.org/W17-3207
2017
Earlier work this paper cites.
R. Shokri, M. Stronati, C. Song, and V. Shmatikov, “Membership inference attacks against machine learning models,” in 2017 IEEE Symposium on Security and Privacy (SP) , 2017, pp. 3–18
2017
Earlier work this paper cites.
B. Hitaj, G. Ateniese, and F. Perez-Cruz, “Deep models under the gan: Information leakage from collaborative deep learning,” in Proceedings of the 2017 ACM SIGSAC Conference on Computer and Communications Security , ser. CCS ’17. New York, NY, USA: Association for Computing Machinery, 2017, p. 603–618. [Online]. Available: https://doi.org/10.1145/3133956.3134012
2017
Earlier work this paper cites.
——, “UCI machine learning repository,” 2017. [Online]. Available: http://archive.ics.uci.edu/ml
2017
Earlier work this paper cites.
N. N. Tran and J. Lee, “Online reviews as health data: Examining the association between availability of health care services and patient star ratings exemplified by the yelp academic dataset,” JMIR Public Health Surveill , vol. 3, no. 3, p. e43, 2017
2017
Earlier work this paper cites.
S. Yeom, I. Giacomelli, M. Fredrikson, and S. Jha, “Privacy risk in machine learning: Analyzing the connection to overfitting,” in 2018 IEEE 31st Computer Security Foundations Symposium (CSF) , 2018, pp. 268–282
2018
Earlier work this paper cites.
M. Nasr, R. Shokri, and A. Houmansadr, “Machine learning with membership privacy using adversarial regularization,” in Proceedings of the 2018 ACM SIGSAC Conference on Computer and Communications Security , ser. CCS ’18. New York, NY, USA: Association for Computing Machinery, 2018, p. 634–646. [Online]. Available: https://doi.org/10.1145/3243734.3243855
2018
Earlier work this paper cites.
A. Madry, A. Makelov, L. Schmidt, D. Tsipras, and A. Vladu, “Towards deep learning models resistant to adversarial attacks,” in 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 - May 3, 2018, Conference Track Proceedings , 2018
2018
Earlier work this paper cites.
J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “BERT: Pre-training of deep bidirectional transformers for language understanding,” in Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) . Minneapolis, Minnesota: Association for Computational Linguistics, Jun. 2019, pp. 4171–4186. [Online]. Available: https://aclanthology.org/N19-1423
2019
Earlier work this paper cites.
H. Husain, H.-H. Wu, T. Gazit, M. Allamanis, and M. Brockschmidt, “CodeSearchNet challenge: Evaluating the state of semantic code search,” arXiv preprint arXiv:1909.09436 , 2019
Original
2019
Earlier work this paper cites.
A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, and I. Sutskever, “Language models are unsupervised multitask learners,” 2019
2019
Earlier work this paper cites.
S. Truex, L. Liu, M. E. Gursoy, L. Yu, and W. Wei, “Demystifying membership inference attacks in machine learning as a service,” IEEE Transactions on Services Computing , vol. 14, no. 6, pp. 2073–2089, 2019
2019
Earlier work this paper cites.
K. Krishna, G. S. Tomar, A. P. Parikh, N. Papernot, and M. Iyyer, “Thieves on sesame street! model extraction of bert-based apis,” arXiv preprint arXiv:1910.12366 , 2019
Original
2019
Earlier work this paper cites.
M. Nasr, R. Shokri, and A. Houmansadr, “Comprehensive privacy analysis of deep learning: Passive and active white-box inference attacks against centralized and federated learning,” in 2019 IEEE symposium on security and privacy (SP) . IEEE, 2019, pp. 739–753
2019
Earlier work this paper cites.
C. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y. Zhou, W. Li, and P. J. Liu, “Exploring the limits of transfer learning with a unified text-to-text transformer,” Journal of Machine Learning Research , vol. 21, no. 140, pp. 1–67, 2020. [Online]. Available: http://jmlr.org/papers/v21/20-074.html
2020
Earlier work this paper cites.

Similar

N. Yefet, U. Alon, and E. Yahav, “Adversarial examples for models of code,” Proc. ACM Program. Lang. , vol. 4, no. OOPSLA, pp. 162:1–162:30, 2020
2020
Cited alongside, same era.
S. Hisamoto, M. Post, and K. Duh, “Membership inference attacks on sequence-to-sequence models: Is my data in your machine translation system?” Transactions of the Association for Computational Linguistics , vol. 8, pp. 49–63, 2020. [Online]. Available: https://aclanthology.org/2020.tacl-1.4
2020
Cited alongside, same era.
Z. Feng, D. Guo, D. Tang, N. Duan, X. Feng, M. Gong, L. Shou, B. Qin, T. Liu, D. Jiang, and M. Zhou, “CodeBERT: A pre-trained model for programming and natural languages,” in Findings of the Association for Computational Linguistics: EMNLP 2020 . Association for Computational Linguistics, Nov. 2020, pp. 1536–1547
2020
Cited alongside, same era.
H. Zhang, Z. Li, G. Li, L. Ma, Y. Liu, and Z. Jin, “Generating adversarial examples for holding robustness of source code processing models,” Proceedings of the AAAI Conference on Artificial Intelligence , vol. 34, no. 01, pp. 1169–1176, Apr. 2020
2020
Cited alongside, same era.
R. Karampatsis, H. Babii, R. Robbes, C. Sutton, and A. Janes, “Big code != big vocabulary: Open-vocabulary models for source code,” in 2020 IEEE/ACM 42nd International Conference on Software Engineering (ICSE) . Los Alamitos, CA, USA: IEEE Computer Society, oct 2020, pp. 1073–1085. [Online]. Available: https://doi.ieeecomputersociety.org/
2020
Cited alongside, same era.
S. Yan, G. Tao, X. Liu, J. Zhai, S. Ma, L. Xu, and X. Zhang, “Correlations between deep neural network model coverage criteria and model quality,” in Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering , ser. ESEC/FSE 2020. New York, NY, USA: Association for Computing Machinery, 2020, p. 775–787. [Online]. Available: https://doi.org/10.1145/3368089.3409671
2020
Cited alongside, same era.
D. Chen, N. Yu, Y. Zhang, and M. Fritz, “Gan-leaks: A taxonomy of membership inference attacks against generative models,” in Proceedings of the 2020 ACM SIGSAC Conference on Computer and Communications Security , ser. CCS ’20. New York, NY, USA: Association for Computing Machinery, 2020, p. 343–362. [Online]. Available: https://doi.org/10.1145/3372297.3417238
2020
Cited alongside, same era.
W. Ahmad, S. Chakraborty, B. Ray, and K.-W. Chang, “Unified pre-training for program understanding and generation,” in Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies . Online: Association for Computational Linguistics, Jun. 2021, pp. 2655–2668
2021
Cited alongside, same era.
S. Lu, D. Guo, S. Ren, J. Huang, A. Svyatkovskiy, A. Blanco, C. B. Clement, D. Drain, D. Jiang, D. Tang, G. Li, L. Zhou, L. Shou, L. Zhou, M. Tufano, M. Gong, M. Zhou, N. Duan, N. Sundaresan, S. K. Deng, S. Fu, and S. Liu, “Codexglue: A machine learning benchmark dataset for code understanding and generation,” CoRR , 2021
2021
Cited alongside, same era.
A. Mazuera-Rozo, A. Mojica-Hanke, M. Linares-Vásquez, and G. Bavota, “Shallow or deep? an empirical study on detecting vulnerabilities using deep learning,” in 2021 IEEE/ACM 29th International Conference on Program Comprehension (ICPC) , 2021, pp. 276–287
2021
Cited alongside, same era.
M. Chen, J. Tworek, and H. J. et al., “Evaluating large language models trained on code,” CoRR , 2021
2021
Cited alongside, same era.
S. Srikant, S. Liu, T. Mitrovska, S. Chang, Q. Fan, G. Zhang, and U. O’Reilly, “Generating Adversarial Computer Programs using Optimized Obfuscations,” ICLR , vol. 16, pp. 209–226, 2021
2021
Cited alongside, same era.

Then

M. L. Siddiq, S. H. Majumder, M. R. Mim, S. Jajodia, and J. C. S. Santos, “An empirical study of code smells in transformer-based code generation techniques,” in 2022 IEEE 22nd International Working Conference on Source Code Analysis and Manipulation (SCAM) , 2022, pp. 71–82
2022
Later among the works it cites.
Z. Sun, X. Du, F. Song, M. Ni, and L. Li, “Coprotector: Protect open-source code against unauthorized training usage with data poisoning,” in Proceedings of the ACM Web Conference 2022 , ser. WWW ’22. New York, NY, USA: Association for Computing Machinery, 2022, p. 652–660
2022
Later among the works it cites.
F. F. Xu, U. Alon, G. Neubig, and V. J. Hellendoorn, “A systematic evaluation of large language models of code,” in Proceedings of the 6th ACM SIGPLAN International Symposium on Machine Programming , ser. MAPS 2022. New York, NY, USA: Association for Computing Machinery, 2022, p. 1–10. [Online]. Available: https://doi.org/10.1145/3520312.3534862
2022
Later among the works it cites.
H. Ye, M. Martinez, and M. Monperrus, “Neural program repair with execution-based backpropagation,” in Proceedings of the 44th International Conference on Software Engineering , ser. ICSE ’22. New York, NY, USA: Association for Computing Machinery, 2022, p. 1506–1518. [Online]. Available: https://doi.org/10.1145/3510003.3510222
2022
Later among the works it cites.
X. Zheng and J. Jiang, “An empirical study of memorization in NLP,” in Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) . Dublin, Ireland: Association for Computational Linguistics, May 2022, pp. 6265–6278. [Online]. Available: https://aclanthology.org/2022.acl-long.434
2022
Later among the works it cites.
H. Pearce, B. Ahmad, B. Tan, B. Dolan-Gavitt, and R. Karri, “Asleep at the keyboard? assessing the security of github copilot’s code contributions,” in 43rd IEEE Symposium on Security and Privacy, SP 2022, San Francisco, CA, USA, May 22-26, 2022 . IEEE, 2022, pp. 754–768. [Online]. Available: https://doi.org/10.1109/SP46214.2022.9833571
2022
Later among the works it cites.
Z. Zeng, H. Tan, H. Zhang, J. Li, Y. Zhang, and L. Zhang, “An extensive study on pre-trained models for program understanding and generation,” in Proceedings of the 31st ACM SIGSOFT International Symposium on Software Testing and Analysis , ser. ISSTA 2022. New York, NY, USA: Association for Computing Machinery, 2022, p. 39–51. [Online]. Available: https://doi.org/10.1145/3533767.3534390
2022
Later among the works it cites.
J. He, B. Xu, Z. Yang, D. Han, C. Yang, and D. Lo, “Ptm4tag: Sharpening tag recommendation of stack overflow posts with pre-trained models,” ser. ICPC ’22. New York, NY, USA: Association for Computing Machinery, 2022, p. 1–11. [Online]. Available: https://doi.org/10.1145/3524610.3527897
2022
Later among the works it cites.
X. Tang, S. Mahloujifar, L. Song, V. Shejwalkar, M. Nasr, A. Houmansadr, and P. Mittal, “Mitigating membership inference attacks by { \{ Self-Distillation } \} through a novel ensemble architecture,” in 31st USENIX Security Symposium (USENIX Security 22) , 2022, pp. 1433–1450
2022
Later among the works it cites.
M. Wei, Y. Huang, J. Yang, J. Wang, and S. Wang, “Cocofuzzing: Testing neural code models with coverage-guided fuzzing,” IEEE Transactions on Reliability , pp. 1–14, 2022
2022
Later among the works it cites.
G. Ramakrishnan and A. Albarghouthi, “Backdoors in neural models of source code,” in 2022 26th International Conference on Pattern Recognition (ICPR) . Los Alamitos, CA, USA: IEEE Computer Society, aug 2022, pp. 2892–2899. [Online]. Available: https://doi.ieeecomputersociety.org/10.1109/ICPR56361.2022.9956690
2022
Later among the works it cites.
J. Li, Z. Li, H. Zhang, G. Li, Z. Jin, X. Hu, and X. Xia, “Poison attack and defense on deep source code processing models,” 2022. [Online]. Available: https://arxiv.org/abs/2210.17029
Original
2022
Later among the works it cites.
F. Mireshghallah, K. Goyal, A. Uniyal, T. Berg-Kirkpatrick, and R. Shokri, “Quantifying privacy risks of masked language models using membership inference attacks,” in Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing . Abu Dhabi, United Arab Emirates: Association for Computational Linguistics, Dec. 2022, pp. 8332–8347. [Online]. Available: https://aclanthology.org/2022.emnlp-main.570
2022
Later among the works it cites.
X. Hou, Y. Zhao, Y. Liu, Z. Yang, K. Wang, L. Li, X. Luo, D. Lo, J. Grundy, and H. Wang, “Large language models for software engineering: A systematic literature review,” 2023
2023
Closest in time.
C. Yang, B. Xu, F. Thung, Y. Shi, T. Zhang, Z. Yang, X. Zhou, J. Shi, J. He, D. Han, and D. Lo, Answer Summarization for Technical Queries: Benchmark and New Approach . New York, NY, USA: Association for Computing Machinery, 2023. [Online]. Available: https://doi.org/10.1145/3551349.3560421
2023
Closest in time.
T.-D. Nguyen, Z. Yang, X. B. D. Le, Patanamon, Thongtanunam, and D. Lo, “Adversarial attacks on code models with discriminative graph patterns,” 2023
2023
Closest in time.
L. Niu, S. Mirza, Z. Maradni, and C. Pöpper, “CodexLeaks: Privacy leaks from code generation language models in GitHub copilot,” in 32nd USENIX Security Symposium (USENIX Security 23) . Anaheim, CA: USENIX Association, Aug. 2023, pp. 2133–2150
2023
Closest in time.
“Aws codewhisperer: Features,” https://aws.amazon.com/codewhisperer/features/ , accessed: March 29, 2023
2023
Closest in time.
S. K. Basak, L. Neil, B. Reaves, and L. Williams, “Secretbench: A dataset of software secrets,” in Proceedings of the 20th International Conference on Mining Software Repositories , ser. MSR ’23, 2023
2023
Closest in time.
E. Nijkamp, B. Pang, H. Hayashi, L. Tu, H. Wang, Y. Zhou, S. Savarese, and C. Xiong, “Codegen: An open large language model for code with multi-turn program synthesis,” in The Eleventh International Conference on Learning Representations , 2023
2023
Closest in time.
J. He, Z. Xin, B. Xu, T. Zhang, K. Kim, Z. Yang, F. Thung, I. Irsan, and D. Lo, “Representation learning for stack overflow posts: How far are we?” 2023
2023
Closest in time.
K. Jesse, T. Ahmed, P. T. Devanbu, and E. Morgan, “Large language models and simple, stupid bugs,” in 2023 IEEE/ACM 20th International Conference on Mining Software Repositories (MSR) . Los Alamitos, CA, USA: IEEE Computer Society, may 2023, pp. 563–575. [Online]. Available: https://doi.ieeecomputersociety.org/10.1109/MSR59073.2023.00082
2023
Closest in time.
“A massively spiffy yet delicately unobtrusive compression library,” https://zlib.net/ , accessed on March 27, 2023
2023
Closest in time.
D. Cotroneo, C. Improta, P. Liguori, and R. Natella, “Vulnerabilities in ai code generators: Exploring targeted data poisoning attacks,” 2023
2023
Closest in time.
T. T. sitter Contributors, “Tree-sitter: An A incremental parsing library,” https://tree-sitter.github.io/tree-sitter/ , Accessed on 25th March, 2023
2023
Closest in time.
Z. Yang, J. Shi, M. H. Asyrofi, B. Xu, X. Zhou, D. Han, and D. Lo, “Prioritizing speech test cases,” 2023. [Online]. Available: https://arxiv.org/abs/2302.00330
Original
2023
Closest in time.
Z. Yang, C. Wang, J. Shi, T. Hoang, P. Kochhar, Q. Lu, Z. Xing, and D. Lo, “What do users ask in open-source ai repositories? an empirical study of github issues,” in Proceedings of the 20th International Conference on Mining Software Repositories , ser. MSR ’23, 2023
2023
Closest in time.
A. Jha and C. K. Reddy, “Codeattack: code-based adversarial attacks for pre-trained programming language models,” in Proceedings of the Thirty-Seventh AAAI Conference on Artificial Intelligence and Thirty-Fifth Conference on Innovative Applications of Artificial Intelligence and Thirteenth Symposium on Educational Advances in Artificial Intelligence , ser. AAAI’23/IAAI’23/EAAI’23. AAAI Press, 2023. [Online]. Available: https://doi.org/10.1609/aaai.v37i12.26739
2023
Closest in time.
D. Dua and C. Graff, “Uci machine learning repository,” http://archive.ics.uci.edu/ml , 2017, accessed: 2023-03-25
2023
Closest in time.
Z. Yang, Z. Zhao, C. Wang, J. Shi, D. Kim, D. Han, and D. Lo, “Unveiling memorization in code models,” in Proceedings of the IEEE/ACM 46th International Conference on Software Engineering , ser. ICSE ’24. New York, NY, USA: Association for Computing Machinery, 2024. [Online]. Available: https://doi.org/10.1145/3597503.3639074
2024
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…