SCGAgent: Recreating the Benefits of Reasoning Models for Secure Code Generation with Agentic Workflows

SEI CERT, “Sei CERT c coding standard,” https://wiki.sei.cmu.edu/confluence/display/c/SEI+CERT+C+Coding+Standard, Pittsburgh, PA, 2016, version 3.1 (Latest stable release)

2016

Earlier work this paper cites.

J. Li, B. Zhao, and C. Zhang, “Fuzzing: a survey,” Cybersecurity , vol. 1, pp. 1–13, 2018

2018

Earlier work this paper cites.

J. Kaplan, S. McCandlish, T. Henighan, T. B. Brown, B. Chess, R. Child, S. Gray, A. Radford, J. Wu, and D. Amodei, “Scaling laws for neural language models,” arXiv preprint arXiv:2001.08361 , 2020

Original

2020

Earlier work this paper cites.

M. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. de Oliveira Pinto, J. Kaplan, H. Edwards, Y. Burda, N. Joseph, G. Brockman, A. Ray, R. Puri, G. Krueger, M. Petrov, H. Khlaaf, G. Sastry, P. Mishkin, B. Chan, S. Gray, N. Ryder, M. Pavlov, A. Power, L. Kaiser, M. Bavarian, C. Winter, P. Tillet, F. P. Such, D. Cummings, M. Plappert, F. Chantzis, E. Barnes, A. Herbert-Voss, W. H. Guss, A. Nichol, A. Paino, N. Tezak, J. Tang, I. Babuschkin, S. Balaji, S. Jain, W. Saunders, C. Hesse, A. N. Carr, J. Leike, J. Achiam, V. Misra, E. Morikawa, A. Radford, M. Knight, M. Brundage, M. Murati, K. Mayer, P. Welinder, B. McGrew, D. Amodei, S. McCandlish, I. Sutskever, and W. Zaremba, “Evaluating large language models trained on code,” 2021

2021

Earlier work this paper cites.

D. Kocetkov, R. Li, L. B. Allal, J. Li, C. Mou, C. M. Ferrandis, Y. Jernite, M. Mitchell, S. Hughes, T. Wolf, D. Bahdanau, L. von Werra, and H. de Vries, “The stack: 3 tb of permissively licensed source code,” 2022. [Online]. Available: https://arxiv.org/abs/2211.15533

Original

2022

Earlier work this paper cites.

J. Hoffmann, S. Borgeaud, A. Mensch, E. Buchatskaya, T. Cai, E. Rutherford, D. d. L. Casas, L. A. Hendricks, J. Welbl, A. Clark et al. , “Training compute-optimal large language models,” arXiv preprint arXiv:2203.15556 , 2022

Original

2022

Earlier work this paper cites.

H. Pearce, B. Ahmad, B. Tan, B. Dolan-Gavitt, and R. Karri, “Asleep at the keyboard? assessing the security of github copilot’s code contributions,” in 2022 IEEE Symposium on Security and Privacy (SP) , 2022, pp. 754–768

2022

Earlier work this paper cites.

M. L. Siddiq and J. C. S. Santos, “Securityeval dataset: mining vulnerability examples to evaluate machine learning-based code generation techniques,” ser. MSR4P&S 2022. New York, NY, USA: Association for Computing Machinery, 2022, p. 29–33. [Online]. Available: https://doi.org/10.1145/3549035.3561184

2022

Earlier work this paper cites.

M. Bhatt, S. Chennabasappa, C. Nikolaidis, S. Wan, I. Evtimov, D. Gabi, D. Song, F. Ahmad, C. Aschermann, L. Fontana, S. Frolov, R. P. Giri, D. Kapil, Y. Kozyrakis, D. LeBlanc, J. Milazzo, A. Straumann, G. Synnaeve, V. Vontimitta, S. Whitman, and J. Saxe, “Purple llama cyberseceval: A secure coding benchmark for language models,” 2023. [Online]. Available: https://arxiv.org/abs/2312.04724

Original

2023

Earlier work this paper cites.

N. Tihanyi, T. Bisztray, R. Jain, M. A. Ferrag, L. C. Cordeiro, and V. Mavroeidis, “The formai dataset: Generative ai in software security through the lens of formal verification,” in Proceedings of the 19th International Conference on Predictive Models and Data Analytics in Software Engineering , ser. PROMISE 2023. New York, NY, USA: Association for Computing Machinery, 2023, p. 33–43. [Online]. Available: https://doi.org/10.1145/3617555.3617874

2023

Earlier work this paper cites.

R. Li, L. B. Allal, Y. Zi, N. Muennighoff, D. Kocetkov, C. Mou, M. Marone, C. Akiki, J. Li, J. Chim, Q. Liu, E. Zheltonozhskii, T. Y. Zhuo, T. Wang, O. Dehaene, M. Davaadorj, J. Lamy-Poirier, J. Monteiro, O. Shliazhko, N. Gontier, N. Meade, A. Zebaze, M.-H. Yee, L. K. Umapathi, J. Zhu, B. Lipkin, M. Oblokulov, Z. Wang, R. Murthy, J. Stillerman, S. S. Patel, D. Abulkhanov, M. Zocca, M. Dey, Z. Zhang, N. Fahmy, U. Bhattacharyya, W. Yu, S. Singh, S. Luccioni, P. Villegas, M. Kunakov, F. Zhdanov, M. Romero, T. Lee, N. Timor, J. Ding, C. Schlesinger, H. Schoelkopf, J. Ebert, T. Dao, M. Mishra, A. Gu, J. Robinson, C. J. Anderson, B. Dolan-Gavitt, D. Contractor, S. Reddy, D. Fried, D. Bahdanau, Y. Jernite, C. M. Ferrandis, S. Hughes, T. Wolf, A. Guha, L. von Werra, and H. de Vries, “Starcoder: may the source be with you!” 2023. [Online]. Available: https://arxiv.org/abs/2305.06161

Original

2023

Earlier work this paper cites.

Z. Luo, C. Xu, P. Zhao, Q. Sun, X. Geng, W. Hu, C. Tao, J. Ma, Q. Lin, and D. Jiang, “Wizardcoder: Empowering code large language models with evol-instruct,” arXiv preprint arXiv:2306.08568 , 2023

Original

2023

Earlier work this paper cites.

S. Chaudhary, “Code alpaca: An instruction-following llama model for code generation,” https://github.com/sahil280114/codealpaca, 2023

2023

Earlier work this paper cites.

J. He and M. Vechev, “Large language models for code: Security hardening and adversarial testing,” in Proceedings of the 2023 ACM SIGSAC Conference on Computer and Communications Security , ser. CCS ’23. New York, NY, USA: Association for Computing Machinery, 2023, p. 1865–1879. [Online]. Available: https://doi.org/10.1145/3576915.3623175

2023

Earlier work this paper cites.

D. Fried, A. Aghajanyan, J. Lin, S. Wang, E. Wallace, F. Shi, R. Zhong, W. tau Yih, L. Zettlemoyer, and M. Lewis, “Incoder: A generative model for code infilling and synthesis,” 2023. [Online]. Available: https://arxiv.org/abs/2204.05999

Original

2023

Earlier work this paper cites.

T. Schick, J. Dwivedi-Yu, R. Dessí, R. Raileanu, M. Lomeli, E. Hambro, L. Zettlemoyer, N. Cancedda, and T. Scialom, “Toolformer: language models can teach themselves to use tools,” in Proceedings of the 37th International Conference on Neural Information Processing Systems , ser. NIPS ’23. Red Hook, NY, USA: Curran Associates Inc., 2023

2023

Earlier work this paper cites.

OpenAI, “Openai o1 system card,” 2024. [Online]. Available: https://arxiv.org/abs/2412.16720

Original

2024

Earlier work this paper cites.

S. Pichai, “Q3 earnings call: CEO’s remarks,” https://blog.google/inside-google/message-ceo/alphabet-earnings-q3-2024/

2024

Cited alongside, same era.

V. Majdinasab, M. J. Bishop, S. Rasheed, A. Moradidakhel, A. Tahir, and F. Khomh, “ Assessing the Security of GitHub Copilot’s Generated Code - A Targeted Replication Study ,” in 2024 IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER) . Los Alamitos, CA, USA: IEEE Computer Society, Mar. 2024, pp. 435–444. [Online]. Available: https://doi.ieeecomputersociety.org/10.1109/SANER60148.2024.00051

2024

Cited alongside, same era.

Y. Yang, Y. Nie, Z. Wang, Y. Tang, W. Guo, B. Li, and D. Song, “Seccodeplt: A unified platform for evaluating the security of code genai,” 2024. [Online]. Available: https://arxiv.org/abs/2410.11096

Original

2024

Cited alongside, same era.

C. Tony, N. E. D. Ferreyra, M. Mutas, S. Dhiff, and R. Scandariato, “Prompting techniques for secure code generation: A systematic investigation,” ArXiv , vol. abs/2407.07064, 2024. [Online]. Available: https://api.semanticscholar.org/CorpusID:271064536

J. Yang, C. E. Jimenez, A. Wettig, K. Lieret, S. Yao, K. Narasimhan, and O. Press, “Swe-agent: Agent-computer interfaces enable automated software engineering,” in Advances in Neural Information Processing Systems , A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomczak, and C. Zhang, Eds., vol. 37. Curran Associates, Inc., 2024, pp. 50 528–50 652. [Online]. Available: https://proceedings.neurips.cc/paper_ files/paper/2024/file/5a7c947568c1b1328ccc5230172e1e7c-Paper-Conference.pdf

2024

Later among the works it cites.

R. Pan, M. Kim, R. Krishna, R. Pavuluri, and S. Sinha, “Multi-language unit test generation using llms,” arXiv preprint arXiv:2409.03093 , 2024

Original

2024

Later among the works it cites.

Z. Yuan, Y. Lou, M. Liu, S. Ding, K. Wang, Y. Chen, and X. Peng, “No more manual tests? evaluating and improving chatgpt for unit test generation,” 2024. [Online]. Available: https://arxiv.org/abs/2305.04207

Original

2024

Later among the works it cites.