Human-Like Code Quality Evaluation through LLM-based Recursive Semantic Comprehension

M. G. Kendall, “A new measure of rank correlation,” Biometrika , vol. 30, no. 1/2, pp. 81–93, 1938

1938

Earlier work this paper cites.

K. Papineni, S. Roukos, T. Ward, and W.-J. Zhu, “Bleu: a method for automatic evaluation of machine translation,” in Proceedings of the 40th annual meeting of the Association for Computational Linguistics , 2002, pp. 311–318

2002

Earlier work this paper cites.

L. C. ROUGE, “A package for automatic evaluation of summaries,” in Proceedings of Workshop on Text Summarization of ACL, Spain , vol. 5, 2004

2004

Earlier work this paper cites.

S. Banerjee and A. Lavie, “Meteor: An automatic metric for mt evaluation with improved correlation with human judgments,” in Proceedings of the acl workshop on intrinsic and extrinsic evaluation measures for machine translation and/or summarization , 2005, pp. 65–72

2005

Earlier work this paper cites.

I. Cohen, Y. Huang, J. Chen, J. Benesty, J. Benesty, J. Chen, Y. Huang, and I. Cohen, “Pearson correlation coefficient,” Noise reduction in speech processing , pp. 1–4, 2009

2009

Earlier work this paper cites.

M. Popović, “chrf: character n-gram f-score for automatic mt evaluation,” in Proceedings of the tenth workshop on statistical machine translation , 2015, pp. 392–395

2015

Earlier work this paper cites.

A. Joshi, S. Kale, S. Chandel, and D. K. Pal, “Likert scale: Explored and explained,” British journal of applied science & technology , vol. 7, no. 4, pp. 396–403, 2015

2015

Earlier work this paper cites.

W. Ling, E. Grefenstette, K. M. Hermann, T. Kočiskỳ, A. Senior, F. Wang, and P. Blunsom, “Latent predictor networks for code generation,” arXiv preprint arXiv:1603.06744 , 2016

Original

2016

Earlier work this paper cites.

J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “Bert: Pre-training of deep bidirectional transformers for language understanding,” arXiv preprint arXiv:1810.04805 , 2018

Original

2018

Earlier work this paper cites.

P. Yin, B. Deng, E. Chen, B. Vasilescu, and G. Neubig, “Learning to mine aligned code and natural language pairs from stack overflow,” in Proceedings of the 15th international conference on mining software repositories , 2018, pp. 476–486

2018

Earlier work this paper cites.

S. Kulal, P. Pasupat, K. Chandra, M. Lee, O. Padon, A. Aiken, and P. S. Liang, “Spoc: Search-based pseudocode to code,” Advances in Neural Information Processing Systems , vol. 32, 2019

2019

Earlier work this paper cites.

N. Tran, H. Tran, S. Nguyen, H. Nguyen, and T. Nguyen, “Does bleu score work for code migration?” in 2019 IEEE/ACM 27th International Conference on Program Comprehension (ICPC) . IEEE, 2019, pp. 165–176

2019

Earlier work this paper cites.

Zhou, Shuyan and Alon, Uri and Agarwal, Sumit and Neubig, Graham, “Codebleu: a method for automatic evaluation of code synthesis,” arXiv preprint arXiv:2009.10297 , 2020

Original

2020

Earlier work this paper cites.

P. R. Nair, “Increasing employability of indian engineering graduates through experiential learning programs and competitive programming: Case study,” Procedia Computer Science , vol. 172, pp. 831–837, 2020

2020

Earlier work this paper cites.

T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell et al. , “Language models are few-shot learners,” Advances in neural information processing systems , vol. 33, pp. 1877–1901, 2020

2020

Earlier work this paper cites.

B. Roziere, M.-A. Lachaux, L. Chanussot, and G. Lample, “Unsupervised translation of programming languages,” Advances in Neural Information Processing Systems , vol. 33, pp. 20 601–20 611, 2020

2020

Cited alongside, same era.

S. Lu, D. Guo, S. Ren, J. Huang, A. Svyatkovskiy, A. Blanco, C. Clement, D. Drain, D. Jiang, D. Tang et al. , “Codexglue: A machine learning benchmark dataset for code understanding and generation,” arXiv preprint arXiv:2102.04664 , 2021

Original

2021

Cited alongside, same era.

D. Hendrycks, S. Basart, S. Kadavath, M. Mazeika, A. Arora, E. Guo, C. Burns, S. Puranik, H. He, D. Song et al. , “Measuring coding challenge competence with apps,” arXiv preprint arXiv:2105.09938 , 2021

Original

2021

Cited alongside, same era.

R. Puri, D. S. Kung, G. Janssen, W. Zhang, G. Domeniconi, V. Zolotov, J. Dolby, J. Chen, M. Choudhury, L. Decker et al. , “Codenet: A large-scale ai for code dataset for learning a diversity of coding tasks,” arXiv preprint arXiv:2105.12655 , 2021

Original

2021

Cited alongside, same era.

Y. Dong, J. Ding, X. Jiang, Z. Li, G. Li, and Z. Jin, “Codescore: Evaluating code generation by learning code execution,” arXiv preprint arXiv:2301.09043 , 2023

Original

2023

Later among the works it cites.

M. Evtikhiev, E. Bogomolov, Y. Sokolov, and T. Bryksin, “Out of the bleu: how should we assess quality of the code generation models?” Journal of Systems and Software , vol. 203, p. 111741, 2023

2023

Later among the works it cites.

S. Ren, D. Guo, S. Lu, L. Zhou, S. Liu, D. Tang, N. Sundaresan, M. Zhou, A. Blanco, and S. Ma, “Codebertscore: Evaluating code generation with pretrained models of code,” arXiv preprint arXiv:2302.05527 , 2023

Original

2023

Later among the works it cites.

T. Y. Zhuo, “Large language models are state-of-the-art evaluators of code generation,” arXiv preprint arXiv:2304.14317 , 2023

Original

2023

Later among the works it cites.