Fixing Large Language Models' Specification Misunderstanding for Better Code Generation

F. Wilcoxon, S. Katti, R. A. Wilcox et al. , “Critical values and probability levels for the wilcoxon rank sum test and the wilcoxon signed rank test,” Selected tables in mathematical statistics , vol. 1, pp. 171–259, 1970

1970

Earlier work this paper cites.

K. Beck, Test-driven development: by example . Addison-Wesley Professional, 2003

2003

Earlier work this paper cites.

S. Gulwani, “Programming by examples: Applications, algorithms, and ambiguity resolution,” in Automated Reasoning: 8th International Joint Conference, IJCAR 2016, Coimbra, Portugal, June 27–July 2, 2016, Proceedings 8 . Springer, 2016, pp. 9–14

2016

Earlier work this paper cites.

T. Wolf, L. Debut, V. Sanh, J. Chaumond, C. Delangue, A. Moi, P. Cistac, T. Rault, R. Louf, M. Funtowicz et al. , “Huggingface’s transformers: State-of-the-art natural language processing,” arXiv preprint arXiv:1910.03771 , 2019

Original

2019

Earlier work this paper cites.

S. Kulal, P. Pasupat, K. Chandra, M. Lee, O. Padon, A. Aiken, and P. S. Liang, “Spoc: Search-based pseudocode to code,” Advances in Neural Information Processing Systems , vol. 32, 2019

2019

Earlier work this paper cites.

S. Ren, D. Guo, S. Lu, L. Zhou, S. Liu, D. Tang, N. Sundaresan, M. Zhou, A. Blanco, and S. Ma, “Codebleu: a method for automatic evaluation of code synthesis,” arXiv preprint arXiv:2009.10297 , 2020

Original

2020

Earlier work this paper cites.

K. Gupta, P. E. Christensen, X. Chen, and D. Song, “Synthesize, execute and debug: Learning to repair for neural program synthesis,” Advances in Neural Information Processing Systems , vol. 33, pp. 17 685–17 695, 2020

2020

Earlier work this paper cites.

D. Hendrycks, S. Basart, S. Kadavath, M. Mazeika, A. Arora, E. Guo, C. Burns, S. Puranik, H. He, D. Song et al. , “Measuring coding challenge competence with apps,” arXiv preprint arXiv:2105.09938 , 2021

Original

2021

Earlier work this paper cites.

M. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. d. O. Pinto, J. Kaplan, H. Edwards, Y. Burda, N. Joseph, G. Brockman et al. , “Evaluating large language models trained on code,” arXiv preprint arXiv:2107.03374 , 2021

Original

2021

Earlier work this paper cites.

S. Lu, D. Guo, S. Ren, J. Huang, A. Svyatkovskiy, A. Blanco, C. Clement, D. Drain, D. Jiang, D. Tang et al. , “Codexglue: A machine learning benchmark dataset for code understanding and generation,” arXiv preprint arXiv:2102.04664 , 2021

Original

2021

Earlier work this paper cites.

OpenAI, “Chatgpt: Optimizing language models for dialogue.” https://openai.com/blog/chatgpt , 2022

2022

Earlier work this paper cites.

J. Wei, X. Wang, D. Schuurmans, M. Bosma, F. Xia, E. Chi, Q. V. Le, D. Zhou et al. , “Chain-of-thought prompting elicits reasoning in large language models,” Advances in Neural Information Processing Systems , vol. 35, pp. 24 824–24 837, 2022

2022

Earlier work this paper cites.

B. Chen, F. Zhang, A. Nguyen, D. Zan, Z. Lin, J.-G. Lou, and W. Chen, “Codet: Code generation with generated tests,” in The Eleventh International Conference on Learning Representations , 2022

2022

Earlier work this paper cites.

T. Ahmed and P. Devanbu, “Few-shot training llms for project-specific code-summarization,” in Proceedings of the 37th IEEE/ACM International Conference on Automated Software Engineering , 2022, pp. 1–5

2022

Earlier work this paper cites.

T. Kojima, S. S. Gu, M. Reid, Y. Matsuo, and Y. Iwasawa, “Large language models are zero-shot reasoners,” Advances in neural information processing systems , vol. 35, pp. 22 199–22 213, 2022

2022

Earlier work this paper cites.

X. Wang, J. Wei, D. Schuurmans, Q. Le, E. Chi, S. Narang, A. Chowdhery, and D. Zhou, “Self-consistency improves chain of thought reasoning in language models,” arXiv preprint arXiv:2203.11171 , 2022

Original

2022

Earlier work this paper cites.

Z. Tian, J. Chen, Q. Zhu, J. Yang, and L. Zhang, “Learning to construct better mutation faults,” in Proceedings of the 37th IEEE/ACM International Conference on Automated Software Engineering , 2022, pp. 1–13

2022

Earlier work this paper cites.

A. Eghbali and M. Pradel, “Crystalbleu: precisely and efficiently measuring the similarity of code,” in Proceedings of the 37th IEEE/ACM International Conference on Automated Software Engineering , 2022, pp. 1–12

2022

Earlier work this paper cites.

H. Le, Y. Wang, A. D. Gotmare, S. Savarese, and S. C. H. Hoi, “Coderl: Mastering code generation through pretrained models and deep reinforcement learning,” Advances in Neural Information Processing Systems , vol. 35, pp. 21 314–21 328, 2022

2022

Earlier work this paper cites.

Y. Ma, Y. Yu, S. Li, Y. Jiang, Y. Guo, Y. Zhang, Y. Xie, and X. Liao, “Bridging code semantic and llms: Semantic chain-of-thought prompting for code generation,” arXiv preprint arXiv:2310.10698 , 2023

Original

2023

Earlier work this paper cites.

J. Kaddour, J. Harris, M. Mozes, H. Bradley, R. Raileanu, and R. McHardy, “Challenges and applications of large language models,” arXiv preprint arXiv:2307.10169 , 2023

Original

2023

Cited alongside, same era.

Z. Tian, J. Chen, and Z. Jin, “Code difference guided adversarial example generation for deep code models,” in 2023 38th IEEE/ACM International Conference on Automated Software Engineering (ASE) . IEEE, 2023, pp. 850–862

2023

Cited alongside, same era.

Z. Tian, J. Chen, and X. Zhang, “On-the-fly improving performance of deep code models via input denoising,” in 2023 38th IEEE/ACM International Conference on Automated Software Engineering (ASE) . IEEE, 2023, pp. 560–572

2023

Cited alongside, same era.

Y. Dong, X. Jiang, Z. Jin, and G. Li, “Self-collaboration code generation via chatgpt,” arXiv preprint arXiv:2304.07590 , 2023

Original

2023

Cited alongside, same era.

J. Li, Y. Zhao, Y. Li, G. Li, and Z. Jin, “Acecoder: Utilizing existing code to enhance code generation,” arXiv preprint arXiv:2303.17780 , 2023

Original

2023

Closest in time.

M. Liu, N. Pinckney, B. Khailany, and H. Ren, “Verilogeval: Evaluating large language models for verilog code generation,” in 2023 IEEE/ACM International Conference on Computer Aided Design (ICCAD) . IEEE, 2023, pp. 1–8

2023

Closest in time.

X. Wan, R. Sun, H. Nakhost, H. Dai, J. M. Eisenschlos, S. O. Arik, and T. Pfister, “Universal self-adaptive prompting,” arXiv preprint arXiv:2305.14926 , 2023

Original

2023

Closest in time.

S. Hong, X. Zheng, J. Chen, Y. Cheng, J. Wang, C. Zhang, Z. Wang, S. K. S. Yau, Z. Lin, L. Zhou et al. , “Metagpt: Meta programming for multi-agent collaborative framework,” arXiv preprint arXiv:2308.00352 , 2023

Original

2023

Closest in time.