Decoding Secret Memorization in Code LLMs Through Token-Level Characterization

N. Carlini, S. Chien, M. Nasr, S. Song, A. Terzis, and F. Tramer, “Membership inference attacks from first principles,” in 2022 IEEE Symposium on Security and Privacy (SP) . IEEE, 2022, pp. 1897–1914

1914

Earlier work this paper cites.

P. E. Hart, D. G. Stork, R. O. Duda et al. , Pattern classification . Wiley Hoboken, 2000

2000

Earlier work this paper cites.

C. Dwork, “Differential privacy: A survey of results,” in International conference on theory and applications of models of computation . Springer, 2008, pp. 1–19

Original

2008

Earlier work this paper cites.

M. Riedmiller and A. Lernen, “Multi layer perceptron,” Machine Learning Lab Special Lecture, University of Freiburg , vol. 24, 2014

2014

Earlier work this paper cites.

M. Ott, M. Auli, D. Grangier, and M. Ranzato, “Analyzing uncertainty in neural machine translation,” in International Conference on Machine Learning . PMLR, 2018, pp. 3956–3965

2018

Earlier work this paper cites.

M. Meli, M. R. McNiece, and B. Reaves, “How bad can it git? characterizing secret leakage in public github repositories.” in NDSS , 2019

2019

Earlier work this paper cites.

H. Husain, H.-H. Wu, T. Gazit, M. Allamanis, and M. Brockschmidt, “Codesearchnet challenge: Evaluating the state of semantic code search,” arXiv preprint arXiv:1909.09436 , 2019

Original

2019

Earlier work this paper cites.

N. Carlini, C. Liu, Ú. Erlingsson, J. Kos, and D. Song, “The secret sharer: Evaluating and testing unintended memorization in neural networks,” in 28th USENIX security symposium (USENIX security 19) , 2019, pp. 267–284

2019

Earlier work this paper cites.

Z. Feng, D. Guo, D. Tang, N. Duan, X. Feng, M. Gong, L. Shou, B. Qin, T. Liu, D. Jiang et al. , “Codebert: A pre-trained model for programming and natural languages,” arXiv preprint arXiv:2002.08155 , 2020

Original

2020

Earlier work this paper cites.

V. Feldman, “Does learning require memorization? a short tale about a long tail,” in Proceedings of the 52nd Annual ACM SIGACT Symposium on Theory of Computing , 2020, pp. 954–959

2020

Earlier work this paper cites.

I. Beltagy, M. E. Peters, and A. Cohan, “Longformer: The long-document transformer,” arXiv preprint arXiv:2004.05150 , 2020

Original

2020

Earlier work this paper cites.

M. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. D. O. Pinto, J. Kaplan, H. Edwards, Y. Burda, N. Joseph, G. Brockman et al. , “Evaluating large language models trained on code,” arXiv preprint arXiv:2107.03374 , 2021

Original

2021

Earlier work this paper cites.

S. Lu, D. Guo, S. Ren, J. Huang, A. Svyatkovskiy, A. Blanco, C. Clement, D. Drain, D. Jiang, D. Tang et al. , “Codexglue: A machine learning benchmark dataset for code understanding and generation (2021),” arXiv preprint arXiv:2102.04664

Original

2021

Earlier work this paper cites.

K. Lee, D. Ippolito, A. Nystrom, C. Zhang, D. Eck, C. Callison-Burch, and N. Carlini, “Deduplicating training data makes language models better,” arXiv preprint arXiv:2107.06499 , 2021

Original

2021

Earlier work this paper cites.

N. Carlini, F. Tramer, E. Wallace, M. Jagielski, A. Herbert-Voss, K. Lee, A. Roberts, T. Brown, D. Song, U. Erlingsson et al. , “Extracting training data from large language models,” in 30th USENIX Security Symposium (USENIX Security 21) , 2021, pp. 2633–2650

2021

Earlier work this paper cites.

H.-L. Chiang, K.-C. Chen, W. Rave, M. Khalili Marandi, and G. Fettweis, “Machine-learning beam tracking and weight optimization for mmwave multi-uav links,” IEEE Transactions on Wireless Communications , vol. 20, no. 8, pp. 5481–5494, 2021

2021

Earlier work this paper cites.

F. F. Xu, U. Alon, G. Neubig, and V. J. Hellendoorn, “A systematic evaluation of large language models of code,” in Proceedings of the 6th ACM SIGPLAN International Symposium on Machine Programming , 2022, pp. 1–10

2022

Earlier work this paper cites.

P. Vaithilingam, T. Zhang, and E. L. Glassman, “Expectation vs. experience: Evaluating the usability of code generation tools powered by large language models,” in Chi conference on human factors in computing systems extended abstracts , 2022, pp. 1–7

2022

Earlier work this paper cites.

L. Tunstall, L. Von Werra, and T. Wolf, Natural language processing with transformers . ” O’Reilly Media, Inc.”, 2022

2022

Earlier work this paper cites.

R. Parikh, C. Dupuy, and R. Gupta, “Canary extraction in natural language understanding models,” arXiv preprint arXiv:2203.13920 , 2022

Original

2022

Earlier work this paper cites.

N. Carlini, D. Ippolito, M. Jagielski, K. Lee, F. Tramer, and C. Zhang, “Quantifying memorization across neural language models,” arXiv preprint arXiv:2202.07646 , 2022

Original

2022

Cited alongside, same era.

M. Ciniselli, L. Pascarella, and G. Bavota, “To what extent do deep learning-based code recommenders generate predictions by cloning code from the training set?” in Proceedings of the 19th International Conference on Mining Software Repositories , 2022, pp. 167–178

2022

Cited alongside, same era.

J. Xu, X. Liu, J. Yan, D. Cai, H. Li, and J. Li, “Learning to break the loop: Analyzing and mitigating repetitions for neural text generation,” Advances in Neural Information Processing Systems , vol. 35, pp. 3082–3095, 2022

2022

Cited alongside, same era.

C. Niu, C. Li, V. Ng, D. Chen, J. Ge, and B. Luo, “An empirical comparison of pre-trained models of source code,” in 2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE) . IEEE, 2023, pp. 2136–2148

2023

W. Su, Y. Tang, Q. Ai, C. Wang, Z. Wu, and Y. Liu, “Mitigating entity-level hallucination in large language models,” 2024. [Online]. Available: https://arxiv.org/abs/2407.09417

Original

2024

Closest in time.

S. Gao, W. Mao, C. Gao, L. Li, X. Hu, X. Xia, and M. R. Lyu, “Learning in the wild: Towards leveraging unlabeled data for effectively tuning pre-trained code models,” in Proceedings of the IEEE/ACM 46th International Conference on Software Engineering , 2024, pp. 1–13

2024

Closest in time.

J. Shi, Z. Yang, H. J. Kang, B. Xu, J. He, and D. Lo, “Greening large language models of code,” in Proceedings of the 46th International Conference on Software Engineering: Software Engineering in Society , 2024, pp. 142–153

2024

Closest in time.

Z. Zhou, C. Sha, and X. Peng, “On calibration of pre-trained code models,” in Proceedings of the IEEE/ACM 46th International Conference on Software Engineering , 2024, pp. 1–13

2024

Closest in time.