A Comprehensive Framework for Evaluating API-oriented Code Generation in Large Language Models

Dominance statistics: Ordinal analyses to answer ordinal questions

Norman Cliff. 1993 · 1993

Earlier work this paper cites.

Evaluation metrics for language models

Stanley F Chen, Douglas Beeferman, and Roni Rosenfeld. 1998 · 1998

Earlier work this paper cites.

Qualitative methods in empirical studies of software engineering

Carolyn B. Seaman. 1999 · 1999

Earlier work this paper cites.

A comparison of random forest and its Gini importance with standard chemometric methods for the feature selection and classification of spectral data

Bjoern H Menze, B Michael Kelm, Ralf Masuch, Uwe Himmelreich, Peter Bachert, Wolfgang Petrich, and Fred A Hamprecht. 2009 · 2009

Earlier work this paper cites.

Mining of massive datasets

Anand Rajaraman and Jeffrey D Ullman. 2011 · 2011

Earlier work this paper cites.

Random forests

Adele Cutler, D Richard Cutler, and John R Stevens. 2012 · 2012

Earlier work this paper cites.

Revisiting the impact of classification techniques on the performance of defect prediction models. In 2015 IEEE/ACM 37th IEEE International Conference on Software Engineering , Vol. 1. IEEE, 789–800

Baljinder Ghotra, Shane McIntosh, and Ahmed E Hassan. 2015 · 2015

Earlier work this paper cites.

The impact of using regression models to build defect classifiers. In 2017 IEEE/ACM 14th International Conference on Mining Software Repositories (MSR) . IEEE, 135–145

Gopi Krishnan Rajbahadur, Shaowei Wang, Yasutaka Kamei, and Ahmed E Hassan. 2017 · 2017

Earlier work this paper cites.

AnswerBot: Automated generation of answer summary to developers’ technical questions. In 2017 32nd IEEE/ACM international conference on automated software engineering (ASE) . IEEE, 706–716

Bowen Xu, Zhenchang Xing, Xin Xia, and David Lo. 2017 · 2017

Earlier work this paper cites.

Impact of discretization noise of the dependent variable on machine learning classifiers in software engineering

Gopi Krishnan Rajbahadur, Shaowei Wang, Yasutaka Kamei, and Ahmed E Hassan. 2019 · 2019

Earlier work this paper cites.

How do developers utilize source code from stack overflow?

Yuhao Wu, Shaowei Wang, Cor-Paul Bezemer, and Katsuro Inoue. 2019 · 2019

Earlier work this paper cites.

An empirical study of obsolete answers on stack overflow

Haoxiang Zhang, Shaowei Wang, Tse-Hsun Chen, Ying Zou, and Ahmed E Hassan. 2019 · 2019

Earlier work this paper cites.

Deep learning based software defect prediction

Lei Qiao, Xuesong Li, Qasim Umer, and Ping Guo. 2020 · 2020

Earlier work this paper cites.

Predicting software defects with explainable machine learning. In Proceedings of the XIX Brazilian Symposium on Software Quality . 1–10

Geanderson Santos, Eduardo Figueiredo, Adriano Veloso, Markos Viggiato, and Nivio Ziviani. 2020 · 2020

Earlier work this paper cites.

Evaluating large language models trained on code

Original

Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al · 2021

Earlier work this paper cites.

Retrieval augmented code generation and summarization

Original

Md Rizwan Parvez, Wasi Uddin Ahmad, Saikat Chakraborty, Baishakhi Ray, and Kai-Wei Chang. 2021 · 2021

Earlier work this paper cites.

Codet5: Identifier-aware unified pre-trained encoder-decoder models for code understanding and generation

Original

Yue Wang, Weishi Wang, Shafiq Joty, and Steven CH Hoi. 2021 · 2021

Earlier work this paper cites.

Unixcoder: Unified cross-modal pre-training for code representation

Original

Daya Guo, Shuai Lu, Nan Duan, Yanlin Wang, Ming Zhou, and Jian Yin. 2022 · 2022

Earlier work this paper cites.

Reacc: A retrieval-augmented code completion framework

Original

Shuai Lu, Nan Duan, Hojae Han, Daya Guo, Seung-won Hwang, and Alexey Svyatkovskiy. 2022 · 2022

Earlier work this paper cites.

Asleep at the keyboard? assessing the security of github copilot’s code contributions. In 2022 IEEE Symposium on Security and Privacy (SP) . IEEE, 754–768

Hammond Pearce, Baleegh Ahmad, Benjamin Tan, Brendan Dolan-Gavitt, and Ramesh Karri. 2022 · 2022

Earlier work this paper cites.

Self-consistency improves chain of thought reasoning in language models

Original

Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. 2022 · 2022

Earlier work this paper cites.

CERT: continual pre-training on sketches for library-oriented code generation

Original

Daoguang Zan, Bei Chen, Dejian Yang, Zeqi Lin, Minsu Kim, Bei Guan, Yongji Wang, Weizhu Chen, and Jian-Guang Lou. 2022 · 2022

Earlier work this paper cites.

Source code recommender systems: The practitioners’ perspective. In 2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE) . IEEE, 2161–2172

Matteo Ciniselli, Luca Pascarella, Emad Aghajani, Simone Scalabrino, Rocco Oliveto, and Gabriele Bavota. 2023 · 2023

Earlier work this paper cites.

Github copilot ai pair programmer: Asset or liability?

Arghavan Moradi Dakhel, Vahid Majdinasab, Amin Nikanjam, Foutse Khomh, Michel C Desmarais, and Zhen Ming Jack Jiang. 2023 · 2023

Earlier work this paper cites.

Chain-of-verification reduces hallucination in large language models

Original

Shehzaad Dhuliawala, Mojtaba Komeili, Jing Xu, Roberta Raileanu, Xian Li, Asli Celikyilmaz, and Jason Weston. 2023 · 2023

Earlier work this paper cites.

Self-collaboration code generation via chatgpt

Original

Yihong Dong, Xue Jiang, Zhi Jin, and Ge Li. 2023 · 2023

Earlier work this paper cites.

Classeval: A manually-crafted benchmark for evaluating llms on class-level code generation

Original

Xueying Du, Mingwei Liu, Kaixin Wang, Hanlin Wang, Junwei Liu, Yixuan Chen, Jiayi Feng, Chaofeng Sha, Xin Peng, and Yiling Lou. 2023 · 2023

Earlier work this paper cites.

Large language models for software engineering: Survey and open problems. In 2023 IEEE/ACM International Conference on Software Engineering: Future of Software Engineering (ICSE-FoSE) . IEEE, 31–53

Angela Fan, Beliz Gokkaya, Mark Harman, Mitya Lyubarskiy, Shubho Sengupta, Shin Yoo, and Jie M Zhang. 2023 · 2023

Earlier work this paper cites.

Security weaknesses of copilot generated code in github

Original

Yujia Fu, Peng Liang, Amjed Tahir, Zengyang Li, Mojtaba Shahin, and Jiaxin Yu. 2023 · 2023

Cited alongside, same era.

The impact of prompt engineering in large language model performance: a psychiatric example

Declan Grabb. 2023 · 2023

Cited alongside, same era.

CodeLMSec Benchmark: Systematically Evaluating and Finding Security Vulnerabilities in Black-Box Code Language Models

Original

Hossein Hajipour, Keno Hassler, Thorsten Holz, Lea Schönherr, and Mario Fritz. 2023 · 2023

Cited alongside, same era.

A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questions

Lei Huang, Weitao Ma Weijiang Yu, Weihong Zhong, Zhangyin Feng, Haotian Wang, Qianglong Chen, Weihua Peng, Xiaocheng Feng, and Bing Qinand Ting Liu. 2023 · 2023

Cited alongside, same era.

Survey of Hallucination in Natural Language Generation

Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Ye Jin Bang, Andrea Madotto, and Pascale Fung. 2023 · 2023

Magicoder

[n. d.] · 2024

Closest in time.

Studying and recommending information highlighting in Stack Overflow answers

Shahla Shaan Ahmed, Shaowei Wang, Yuan Tian, Tse-Hsun Peter Chen, and Haoxiang Zhang. 2024 · 2024

Closest in time.

AutoAPIEval

anonymous. 2024 · 2024

Closest in time.

Code search is all you need? improving code suggestions with code search. In Proceedings of the IEEE/ACM 46th International Conference on Software Engineering . 1–13

Junkai Chen, Xing Hu, Zhenhao Li, Cuiyun Gao, Xin Xia, and David Lo. 2024 · 2024

Closest in time.

Exploring RAG-based Vulnerability Augmentation with LLMs

Original

Seyed Shayan Daneshvar, Yu Nong, Xu Yang, Shaowei Wang, and Haipeng Cai. 2024 · 2024

Closest in time.

DeepSeek LLM: Scaling Open-Source Language Models with Longtermism