LLMSecEval: A Dataset of Natural Language Prompts for Security Evaluations

J. L. Fleiss and J. Cohen, “The equivalence of weighted kappa and the intraclass correlation coefficient as measures of reliability.” Educational and Psychological Measurement. , vol. 33(3), pp. 613–619, 1973. [Online]. Available: https://doi.org/10.1177/001316447303300309

1973

Earlier work this paper cites.

M. L. McHugh, “Interrater reliability: The kappa statistic.” Biochem Med (Zagreb). , vol. 22(3), pp. 276–282, 2012

2012

Earlier work this paper cites.

A. Wickert, M. Reif, M. Eichberg, A. Dodhy, and M. Mezini, “A dataset of parametric cryptographic misuses,” in Proceedings of the 16th International Conference on Mining Software Repositories, MSR 2019, 26-27 May 2019, Montreal, Canada , M. D. Storey, B. Adams, and S. Haiduc, Eds. IEEE / ACM, 2019, pp. 96–100. [Online]. Available: https://doi.org/10.1109/MSR.2019.00023

2019

Earlier work this paper cites.

S. E. Ponta, H. Plate, A. Sabetta, M. Bezzi, and C. Dangremont, “A manually-curated dataset of fixes to vulnerabilities of open-source software,” in Proceedings of the 16th International Conference on Mining Software Repositories, MSR 2019, 26-27 May 2019, Montreal, Canada , M. D. Storey, B. Adams, and S. Haiduc, Eds. IEEE / ACM, 2019, pp. 383–387. [Online]. Available: https://doi.org/10.1109/MSR.2019.00064

2019

Earlier work this paper cites.

M. Hazhirpasand, M. Ghafari, and O. Nierstrasz, “Java cryptography uses in the wild,” in ESEM ’20: ACM / IEEE International Symposium on Empirical Software Engineering and Measurement, Bari, Italy, October 5-7, 2020 , M. T. Baldassarre, F. Lanubile, M. Kalinowski, and F. Sarro, Eds. ACM, 2020, pp. 40:1–40:6. [Online]. Available: https://doi.org/10.1145/3382494.3422166

2020

Earlier work this paper cites.

M. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. de Oliveira Pinto, J. Kaplan, H. Edwards, Y. Burda, N. Joseph, G. Brockman, A. Ray, R. Puri, G. Krueger, M. Petrov, H. Khlaaf, G. Sastry, P. Mishkin, B. Chan, S. Gray, N. Ryder, M. Pavlov, A. Power, L. Kaiser, M. Bavarian, C. Winter, P. Tillet, F. P. Such, D. Cummings, M. Plappert, F. Chantzis, E. Barnes, A. Herbert-Voss, W. H. Guss, A. Nichol, A. Paino, N. Tezak, J. Tang, I. Babuschkin, S. Balaji, S. Jain, W. Saunders, C. Hesse, A. N. Carr, J. Leike, J. Achiam, V. Misra, E. Morikawa, A. Radford, M. Knight, M. Brundage, M. Murati, K. Mayer, P. Welinder, B. McGrew, D. Amodei, S. McCandlish, I. Sutskever, and W. Zaremba, “Evaluating large language models trained on code,” CoRR , vol. abs/2107.03374, 2021. [Online]. Available: https://arxiv.org/abs/2107.03374

Original

2021

Earlier work this paper cites.

X. Han, Z. Zhang, N. Ding, Y. Gu, X. Liu, Y. Huo, J. Qiu, Y. Yao, A. Zhang, L. Zhang, W. Han, M. Huang, Q. Jin, Y. Lan, Y. Liu, Z. Liu, Z. Lu, X. Qiu, R. Song, J. Tang, J. Wen, J. Yuan, W. X. Zhao, and J. Zhu, “Pre-trained models: Past, present and future,” AI Open , vol. 2, pp. 225–250, 2021

2021

Cited alongside, same era.

MITRE., “Cwe top 25 most dangerous software weaknesses,” 2021. [Online]. Available: https://cwe.mitre.org/top25/archive/2021/2021_cwe_top25.html

2021

Cited alongside, same era.

J. Austin, A. Odena, M. I. Nye, M. Bosma, H. Michalewski, D. Dohan, E. Jiang, C. J. Cai, M. Terry, Q. V. Le, and C. Sutton, “Program synthesis with large language models,” CoRR , vol. abs/2108.07732, 2021. [Online]. Available: https://arxiv.org/abs/2108.07732

Original

2021

Cited alongside, same era.

L. Reynolds and K. McDonell, “Prompt programming for large language models: Beyond the few-shot paradigm,” in CHI ’21: CHI Conference on Human Factors in Computing Systems, Virtual Event / Yokohama Japan, May 8-13, 2021, Extended Abstracts , Y. Kitamura, A. Quigley, K. Isbister, and T. Igarashi, Eds. ACM, 2021, pp. 314:1–314:7

Original

C. Tony, N. E. Diaz Ferreyra, and R. Scandariato, “Github considered harmful? analyzing open-source projects for the automatic generation of cryptographic api call sequences,” IEEE 22nd International Conference on Software Quality, Reliability and Security , 2022

2022

Later among the works it cites.

E. Kalliamvakou, “Research: Quantifying github copilot’s impact on developer productivity and happiness,” 2022. [Online]. Available: https://github.blog/2022-09-07-research-quantifying-github-copilots-impact-on-developer-productivity-and-happiness/

2022

Later among the works it cites.

H. Pearce, B. Ahmad, B. Tan, B. Dolan-Gavitt, and R. Karri, “Asleep at the keyboard? assessing the security of github copilot’s code contributions,” in 43rd IEEE Symposium on Security and Privacy, SP 2022, San Francisco, CA, USA, May 22-26, 2022 . IEEE, 2022, pp. 754–768. [Online]. Available: https://doi.org/10.1109/SP46214.2022.9833571

2022

Later among the works it cites.