Fetching the paper…

Reading the bibliography…

2024

Reasoning Runtime Behavior of a Program with LLM: How Far Are We?

Chen, Junkai, Pan, Zhiyuan, Hu, Xing et al.

Original
PDF
Post on X

Understand

Large language models for code (i.e., code LLMs) have shown strong code understanding and generation capabilities.

To evaluate the capabilities of code LLMs in various aspects, many benchmarks have been proposed (e.g., HumanEval and ClassEval).
Code reasoning is one of the most essential abilities of code LLMs, but existing benchmarks for code reasoning are not sufficient.
Typically, they focus on predicting the input and output of a program, ignoring the evaluation of the intermediate behavior during program execution, as well as the logical consistency (e.g., the model should not give the correct output if the prediction of execution path is wrong) when performing the reasoning.

Reasoning Runtime Behavior of a Program with LLM: How Far Are We? · Around

Built on

H. Hemmati, “How effective are code coverage criteria?” in 2015 IEEE International Conference on Software Quality, Reliability and Security . IEEE, 2015, pp. 151–156
2015
Earlier work this paper cites.
P. Ammann and J. Offutt, Introduction to software testing . Cambridge University Press, 2016
2016
Earlier work this paper cites.
J. Henkel, S. K. Lahiri, B. Liblit, and T. Reps, “Code vectors: understanding programs through embedded abstracted symbolic traces,” in Proceedings of the 2018 26th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering , ser. ESEC/FSE 2018. New York, NY, USA: Association for Computing Machinery, 2018, p. 163–174. [Online]. Available: https://doi.org/10.1145/3236024.3236085
2018
Earlier work this paper cites.
D. Guo, S. Ren, S. Lu, Z. Feng, D. Tang, S. Liu, L. Zhou, N. Duan, A. Svyatkovskiy, S. Fu et al. , “Graphcodebert: Pre-training code representations with data flow,” arXiv preprint arXiv:2009.08366 , 2020
Original
2020
Earlier work this paper cites.
M. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. d. O. Pinto, J. Kaplan, H. Edwards, Y. Burda, N. Joseph, G. Brockman et al. , “Evaluating large language models trained on code,” arXiv preprint arXiv:2107.03374 , 2021
Original
2021
Earlier work this paper cites.
J. Austin, A. Odena, M. Nye, M. Bosma, H. Michalewski, D. Dohan, E. Jiang, C. Cai, M. Terry, Q. Le et al. , “Program synthesis with large language models,” arXiv preprint arXiv:2108.07732 , 2021
Original
2021
Earlier work this paper cites.
S. Lu, D. Guo, S. Ren, J. Huang, A. Svyatkovskiy, A. Blanco, C. Clement, D. Drain, D. Jiang, D. Tang et al. , “Codexglue: A machine learning benchmark dataset for code understanding and generation,” in Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 1) , 2021
2021
Earlier work this paper cites.
Y. Elazar, N. Kassner, S. Ravfogel, A. Ravichander, E. Hovy, H. Schütze, and Y. Goldberg, “Measuring and improving consistency in pretrained language models,” Transactions of the Association for Computational Linguistics , vol. 9, pp. 1012–1031, 2021
2021
Earlier work this paper cites.
M. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. de Oliveira Pinto, J. Kaplan, H. Edwards, Y. Burda, N. Joseph, G. Brockman, A. Ray, R. Puri, G. Krueger, M. Petrov, H. Khlaaf, G. Sastry, P. Mishkin, B. Chan, S. Gray, N. Ryder, M. Pavlov, A. Power, L. Kaiser, M. Bavarian, C. Winter, P. Tillet, F. P. Such, D. Cummings, M. Plappert, F. Chantzis, E. Barnes, A. Herbert-Voss, W. H. Guss, A. Nichol, A. Paino, N. Tezak, J. Tang, I. Babuschkin, S. Balaji, S. Jain, W. Saunders, C. Hesse, A. N. Carr, J. Leike, J. Achiam, V. Misra, E. Morikawa, A. Radford, M. Knight, M. Brundage, M. Murati, K. Mayer, P. Welinder, B. McGrew, D. Amodei, S. McCandlish, I. Sutskever, and W. Zaremba, “Evaluating large language models trained on code,” 2021
2021
Earlier work this paper cites.
F. Barbieri, L. E. Anke, and J. Camacho-Collados, “Xlm-t: Multilingual language models in twitter for sentiment analysis and beyond,” in Proceedings of the Thirteenth Language Resources and Evaluation Conference , 2022, pp. 258–266
2022
Earlier work this paper cites.
A. Creswell, M. Shanahan, and I. Higgins, “Selection-inference: Exploiting large language models for interpretable logical reasoning,” in The Eleventh International Conference on Learning Representations , 2022
2022
Earlier work this paper cites.
F. Tsimpourlas, G. Rooijackers, A. Rajan, and M. Allamanis, “Embedding and classifying test execution traces using neural networks,” IET Software , vol. 16, no. 3, pp. 301–316, 2022. [Online]. Available: https://ietresearch.onlinelibrary.wiley.com/doi/abs/10.1049/sfw2.12038
2022
Earlier work this paper cites.
M. Jang, D. S. Kwon, and T. Lukasiewicz, “Becel: Benchmark for consistency evaluation of language models,” in Proceedings of the 29th International Conference on Computational Linguistics , 2022, pp. 3680–3696
2022
Earlier work this paper cites.
P. Sahu, M. Cogswell, Y. Gong, and A. Divakaran, “Unpacking large language models with conceptual consistency,” arXiv preprint arXiv:2209.15093 , 2022
Original
2022
Earlier work this paper cites.
X. Wang, J. Wei, D. Schuurmans, Q. V. Le, E. H. Chi, S. Narang, A. Chowdhery, and D. Zhou, “Self-consistency improves chain of thought reasoning in language models,” in The Eleventh International Conference on Learning Representations , 2022
2022
Earlier work this paper cites.
J. Wei, X. Wang, D. Schuurmans, M. Bosma, F. Xia, E. Chi, Q. V. Le, D. Zhou et al. , “Chain-of-thought prompting elicits reasoning in large language models,” Advances in neural information processing systems , vol. 35, pp. 24 824–24 837, 2022
2022
Earlier work this paper cites.
B. Min, H. Ross, E. Sulem, A. P. B. Veyseh, T. H. Nguyen, O. Sainz, E. Agirre, I. Heintz, and D. Roth, “Recent advances in natural language processing via large pre-trained language models: A survey,” ACM Computing Surveys , vol. 56, no. 2, pp. 1–40, 2023
2023
Earlier work this paper cites.
A. Rogers, M. Gardner, and I. Augenstein, “Qa dataset explosion: A taxonomy of nlp resources for question answering and reading comprehension,” ACM Computing Surveys , vol. 55, no. 10, pp. 1–45, 2023
2023
Earlier work this paper cites.
A. Fan, B. Gokkaya, M. Harman, M. Lyubarskiy, S. Sengupta, S. Yoo, and J. M. Zhang, “Large language models for software engineering: Survey and open problems,” arXiv preprint arXiv:2310.03533 , 2023
Original
2023
Earlier work this paper cites.
B. Roziere, J. Gehring, F. Gloeckle, S. Sootla, I. Gat, X. E. Tan, Y. Adi, J. Liu, T. Remez, J. Rapin et al. , “Code llama: Open foundation models for code,” arXiv preprint arXiv:2308.12950 , 2023
Original
2023
Earlier work this paper cites.
Y. Wei, Z. Wang, J. Liu, Y. Ding, and L. Zhang, “Magicoder: Source code is all you need,” arXiv preprint arXiv:2312.02120 , 2023
Original
2023
Earlier work this paper cites.
Y. Chang, X. Wang, J. Wang, Y. Wu, L. Yang, K. Zhu, H. Chen, X. Yi, C. Wang, Y. Wang et al. , “A survey on evaluation of large language models,” ACM Transactions on Intelligent Systems and Technology , 2023
2023
Earlier work this paper cites.

Similar

A. Ni, S. Iyer, D. Radev, V. Stoyanov, W.-t. Yih, S. Wang, and X. V. Lin, “Lever: Learning to verify language-to-code generation with execution,” in International Conference on Machine Learning . PMLR, 2023, pp. 26 106–26 128
2023
Cited alongside, same era.
X. Chen, M. Lin, N. Schärli, and D. Zhou, “Teaching large language models to self-debug,” arXiv preprint arXiv:2304.05128 , 2023
Original
2023
Cited alongside, same era.
M. J. Min, Y. Ding, L. Buratti, S. Pujar, G. Kaiser, S. Jana, and B. Ray, “Beyond accuracy: Evaluating self-consistency of code llms,” in The Twelfth International Conference on Learning Representations , 2023
2023
Cited alongside, same era.
C. Liu, S. Lu, W. Chen, D. Jiang, A. Svyatkovskiy, S. Fu, N. Sundaresan, and N. Duan, “Code execution with pre-trained language models,” arXiv preprint arXiv:2305.05383 , 2023
Original
2023
Cited alongside, same era.

Then

C. Liu, S. D. Zhang, and R. Jabbarvand, “Codemind: A framework to challenge large language models for code reasoning,” arXiv preprint arXiv:2402.09664 , 2024
Original
2024
Closest in time.
Y. Ding, B. Steenhoek, K. Pei, G. Kaiser, W. Le, and B. Ray, “Traced: Execution-aware pre-training for source code,” in Proceedings of the 46th IEEE/ACM International Conference on Software Engineering , 2024, pp. 1–12
2024
Closest in time.
“GitHub Copilot · Your AI pair programmer,” https://github.com/features/copilot , 2024, last accessed Mar. 2024
2024
Closest in time.
M. Allamanis, S. Panthaplackel, and P. Yin, “Unsupervised evaluation of code llms with round-trip correctness,” arXiv preprint arXiv:2402.08699 , 2024
Original
2024
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

E. Jabbar, H. Hemmati, and R. Feldt, “Investigating execution trace embedding for test case prioritization,” in 2023 IEEE 23rd International Conference on Software Quality, Reliability, and Security (QRS) , 2023, pp. 279–290

2023

Cited alongside, same era.

M. Jang and T. Lukasiewicz, “Consistency analysis of chatgpt,” in Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing , 2023, pp. 15 970–15 985

2023

Cited alongside, same era.

H. Touvron, L. Martin, K. Stone, P. Albert, A. Almahairi, Y. Babaei, N. Bashlykov, S. Batra, P. Bhargava, S. Bhosale et al. , “Llama 2: Open foundation and fine-tuned chat models,” arXiv preprint arXiv:2307.09288 , 2023

2023

Cited alongside, same era.

E. Nijkamp, H. Hayashi, C. Xiong, S. Savarese, and Y. Zhou, “Codegen2: Lessons for training llms on programming and natural languages,” arXiv preprint arXiv:2305.02309 , 2023

2023

Cited alongside, same era.

Y. Lai, C. Li, Y. Wang, T. Zhang, R. Zhong, L. Zettlemoyer, W.-t. Yih, D. Fried, S. Wang, and T. Yu, “Ds-1000: A natural and reliable benchmark for data science code generation,” in International Conference on Machine Learning . PMLR, 2023, pp. 18 319–18 345

2023

Cited alongside, same era.

F. Cassano, L. Li, A. Sethi, N. Shinn, A. Brennan-Jones, A. Lozhkov, C. Anderson, and A. Guha, “Can it edit? evaluating the ability of large language models to follow code editing instructions,” arXiv preprint arXiv:2312.12450 , 2023

2023

Cited alongside, same era.

M. A. M. Khan, M. S. Bari, X. L. Do, W. Wang, M. R. Parvez, and S. Joty, “xcodeeval: A large scale multilingual multitask benchmark for code understanding, generation, translation and retrieval,” arXiv preprint arXiv:2303.03004 , 2023

2023

Cited alongside, same era.

M. Tufano, S. Chandel, A. Agarwal, N. Sundaresan, and C. Clement, “Predicting code coverage without execution,” arXiv preprint arXiv:2307.13383 , 2023

Original

2023

Cited alongside, same era.

X. Du, M. Liu, K. Wang, H. Wang, J. Liu, Y. Chen, J. Feng, C. Sha, X. Peng, and Y. Lou, “Classeval: A manually-crafted benchmark for evaluating llms on class-level code generation,” in Proceedings of the 46th IEEE/ACM International Conference on Software Engineering , 2024

2024

Closest in time.

E. La Malfa, C. Weinhuber, O. Torre, F. Lin, A. Cohn, N. Shadbolt, and M. Wooldridge, “Code simulation challenges for large language models,” arXiv preprint arXiv:2401.09074 , 2024

Original

2024

Closest in time.

J. Chen, Z. Li, X. Hu, and X. Xia, “Nlperturbator: Studying the robustness of code llms to natural language variations,” arXiv preprint arXiv:2406.19783 , 2024

2024

Closest in time.

A. Lozhkov, R. Li, L. B. Allal, F. Cassano, J. Lamy-Poirier, N. Tazi et al. , “Starcoder 2 and the stack v2: The next generation,” arXiv preprint arXiv:2402.19173 , 2024

2024

Closest in time.

H. Yu, B. Shen, D. Ran, J. Zhang, Q. Zhang, Y. Ma, G. Liang, Y. Li, Q. Wang, and T. Xie, “Codereval: A benchmark of pragmatic code generation with generative pre-trained models,” in Proceedings of the 46th IEEE/ACM International Conference on Software Engineering , 2024, pp. 1–12

2024

Closest in time.

Y. Ding, Z. Wang, W. Ahmad, H. Ding, M. Tan, N. Jain, M. K. Ramanathan, R. Nallapati, P. Bhatia, D. Roth et al. , “Crosscodeeval: A diverse and multilingual benchmark for cross-file code completion,” Advances in Neural Information Processing Systems , vol. 36, 2024

2024

Closest in time.

J. Li, G. Li, Y. Zhao, Y. Li, Z. Jin, H. Zhu, H. Liu, K. Liu, L. Wang, Z. Fang et al. , “Deveval: Evaluating code generation in practical software projects,” arXiv preprint arXiv:2401.06401 , 2024

2024

Closest in time.

A. Z. Yang, C. Le Goues, R. Martins, and V. Hellendoorn, “Large language models for test-free fault localization,” in Proceedings of the 46th IEEE/ACM International Conference on Software Engineering , 2024, pp. 1–12

2024

Closest in time.

A. G. Shypula, A. Madaan, Y. Zeng, U. Alon, J. R. Gardner, Y. Yang, M. Hashemi, G. Neubig, P. Ranganathan, O. Bastani, and A. Yazdanbakhsh, “Learning performance-improving code edits,” in The Twelfth International Conference on Learning Representations , 2024. [Online]. Available: https://openreview.net/forum?id=ix7rLVHXyY

2024

Closest in time.

J. Chen, X. Hu, Z. Li, C. Gao, X. Xia, and D. Lo, “Code search is all you need? improving code suggestions with code search,” in Proceedings of the 46th IEEE/ACM International Conference on Software Engineering, ICSE 2024, Lisbon, Portugal, April 14-20, 2024 , 2024, pp. 73:1–73:13

2024

Closest in time.

“Models - Hugging Face,” https://huggingface.co/models , 2024, last accessed Mar. 2024

2024

Closest in time.

“API Reference - OpenAI API,” https://platform.openai.com/docs/api-reference , 2024, last accessed Mar. 2024

2024

Closest in time.

“openai_humaneval · Datasets at Hugging Face,” https://huggingface.co/datasets/openai_humaneval , 2024, last accessed Mar. 2024

2024

Closest in time.

“FudanSELab/ClassEval · Datasets at Hugging Face,” https://huggingface.co/datasets/FudanSELab/ClassEval , 2024, last accessed Mar. 2024

2024

Closest in time.

“Replication package,” https://figshare.com/s/e5de95bd79ab5ddea76c , 2024, replication package

2024

Closest in time.

S. Yao, D. Yu, J. Zhao, I. Shafran, T. Griffiths, Y. Cao, and K. Narasimhan, “Tree of thoughts: Deliberate problem solving with large language models,” Advances in Neural Information Processing Systems , vol. 36, 2024

2024

Closest in time.

Y. Peng, C. Gao, Z. Li, B. Gao, D. Lo, Q. Zhang, and M. Lyu, “Static inference meets deep learning: a hybrid type inference approach for python,” in Proceedings of the 44th International Conference on Software Engineering , 2022, pp. 2019–2030

2030

Closest in time.