BaxBench: Can LLMs Generate Correct and Secure Backends?

Abstract interpretation: A unified lattice model for static analysis of programs by construction or approximation of fixpoints

Cousot, P. and Cousot, R · 1977

Earlier work this paper cites.

Evaluating large language models trained on code

Original

Chen, M., Tworek, J., Jun, H., Yuan, Q., de Oliveira Pinto, H. P., Kaplan, J., Edwards, H., Burda, Y., Joseph, N., Brockman, G., et al · 2021

Earlier work this paper cites.

Measuring coding challenge competence with APPS

Hendrycks, D., Basart, S., Kadavath, S., Mazeika, M., Arora, A., Guo, E., Burns, C., Puranik, S., He, H., Song, D., and Steinhardt, J · 2021

Earlier work this paper cites.

Securityeval dataset: Mining vulnerability examples to evaluate machine learning-based code generation techniques

Siddiq, M. L. and Santos, J. C. S · 2022

Earlier work this paper cites.

Purple llama cyberseceval: A secure coding benchmark for language models

Original

Bhatt, M., Chennabasappa, S., Nikolaidis, C., Wan, S., Evtimov, I., Gabi, D., Song, D., Ahmad, F., Aschermann, C., Fontana, L., et al · 2023

Earlier work this paper cites.

DS-1000: A natural and reliable benchmark for data science code generation

Lai, Y., Li, C., Wang, Y., Zhang, T., Zhong, R., Zettlemoyer, L., Yih, W., Fried, D., Wang, S. I., and Yu, T · 2023

Earlier work this paper cites.

Execution-based evaluation for open-domain code generation

Wang, Z., Zhou, S., Fried, D., and Neubig, G · 2023

Earlier work this paper cites.

"false negative-that one is going to kill you": Understanding industry perspectives of static analysis based security testing

Ami, A. S., Moran, K., Poshyvanyk, D., and Nadkarni, A · 2024

Earlier work this paper cites.

The llama 3 herd of models

Original

Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Yang, A., Fan, A., et al · 2024

Earlier work this paper cites.

Constrained decoding for secure code generation

Original

Fu, Y., Baker, E., and Chen, Y · 2024

Earlier work this paper cites.

Redcode: Risky code execution and generation benchmark for code agents

Guo, C., Liu, X., Xie, C., Zhou, A., Zeng, Y., Lin, Z., Song, D., and Li, B · 2024

Earlier work this paper cites.

Codelmsec benchmark: Systematically evaluating and finding security vulnerabilities in black-box code language models

Hajipour, H., Hassler, K., Holz, T., Schönherr, L., and Fritz, M · 2024

Earlier work this paper cites.

Instruction tuning for secure code generation

He, J., Vero, M., Krasnopolska, G., and Vechev, M. T · 2024

Earlier work this paper cites.

Qwen2.5-coder technical report

Original

Hui, B., Yang, J., Cui, Z., Yang, J., Liu, D., Zhang, L., Liu, T., Zhang, J., Yu, B., Lu, K., et al · 2024

Earlier work this paper cites.

Gpt-4o system card

Original

Hurst, A., Lerer, A., Goucher, A. P., Perelman, A., Ramesh, A., Clark, A., Ostrow, A., Welihinda, A., Hayes, A., Radford, A., et al · 2024

Cited alongside, same era.

Openai o1 system card

Original

Jaech, A., Kalai, A., Lerer, A., Richardson, A., El-Kishky, A., Low, A., Helyar, A., Madry, A., Beutel, A., Carney, A., et al · 2024

Cited alongside, same era.

Practical attacks against black-box code completion engines, 2024

Original

Jenko, S., He, J., Mündler, N., Vero, M., and Vechev, M · 2024

Cited alongside, same era.

Swe-bench: Can language models resolve real-world github issues?

Jimenez, C. E., Yang, J., Wettig, A., Yao, S., Pei, K., Press, O., and Narasimhan, K. R · 2024

Cited alongside, same era.

Automatic programming: Large language models and beyond

Lyu, M. R., Ray, B., Roychoudhury, A., Tan, S. H., and Thongtanunam, P · 2024

Bigcodebench: Benchmarking code generation with diverse function calls and complex instructions

Original

Zhuo, T. Y., Vu, M. C., Chim, J., Hu, H., Yu, W., Widyasari, R., Yusuf, I. N. B., Zhan, H., He, J., Paul, I., et al · 2024

Later among the works it cites.

Claude 3.5 sonnet

Anthropic · 2025

Closest in time.

Model card claude 3 addendum

Anthropic · 2025

Closest in time.

Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning

Original

Guo, D., Yang, D., Zhang, H., Song, J., Zhang, R., Xu, R., Zhu, Q., Ma, S., Wang, P., Bi, X., et al · 2025

Closest in time.

Swe-lancer: Can frontier llms earn $1 million from real-world freelance software engineering?

Original

Miserendino, S., Wang, M., Patwardhan, T., and Heidecke, J · 2025

Closest in time.