Fetching the paper…

Reading the bibliography…

2023

Practices and Challenges of Using GitHub Copilot: An Empirical Study

Zhang, Beiqi, Liang, Peng, Zhou, Xiyu et al.

Original
PDF
Post on X

Understand

With the advances in machine learning, there is a growing interest in AI-enabled tools for autocompleting source code.

GitHub Copilot, also referred to as the "AI Pair Programmer", has been trained on billions of lines of open source GitHub code, and is one of such tools that has been increasingly used since its launch on June 2021.
However, little effort has been devoted to understanding the practices and challenges of using Copilot in programming with auto-completed source code.
To this end, we conducted an empirical study by collecting and analyzing the data from Stack Overflow (SO) and GitHub Discussions.

Practices and Challenges of Using GitHub Copilot: An Empirical Study · Around

Built on

J. Cohen, “A coefficient of agreement for nominal scales,” Educational and Psychological Measurement , vol. 20, no. 1, pp. 37–46, 1960
1960
Earlier work this paper cites.
B. G. Glaser, “The constant comparative method of qualitative analysis,” Social Problems , vol. 12, no. 4, pp. 436–445, 1965
1965
Earlier work this paper cites.
L. R. Hallberg, “The “core category” of grounded theory: Making constant comparisons,” International Journal of Qualitative Studies on Health and Well-being , vol. 1, no. 3, pp. 141–148, 2006
2006
Earlier work this paper cites.
A. N. Christopher, Interpreting and Using Statistics in Psychological Research . SAGE Publications, 2017
2017
Earlier work this paper cites.
M. Allamanis, E. T. Barr, P. Devanbu, and C. Sutton, “A survey of machine learning for big code and naturalness,” arXiv preprint abs/1709.06182 , 2018
Original
2018
Earlier work this paper cites.
J. Austin, A. Odena, M. Nye, M. Bosma, H. Michalewski, D. Dohan, E. Jiang, C. Cai, M. Terry, Q. Le et al. , “Program synthesis with large language models,” arXiv preprint abs/2108.07732 , 2021
Original
2021
Earlier work this paper cites.

Similar

H. Pearce, B. Ahmad, B. Tan, B. Dolan-Gavitt, and R. Karri, “An empirical cybersecurity evaluation of github copilot’s code contributions,” arXiv preprint abs/2108.09293 , 2021
Original
2021
Cited alongside, same era.
G. Sandoval, H. Pearce, T. Nys, R. Karri, B. Dolan-Gavitt, and S. Garg, “Security implications of large language model code assistants: A user study,” arXiv preprint abs/2208.09727 , 2022
Original
2022
Cited alongside, same era.
S. Imai, “Is github copilot a substitute for human pair-programming? an empirical study,” in Proc. of the 44th International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) . IEEE, 2022, pp. 319–321
2022
Cited alongside, same era.
B. Yetistiren, I. Ozsoy, and E. Tuzun, “Assessing the quality of github copilot’s code generation,” in Proc. of the 18th International Conference on Predictive Models and Data Analytics in Software Engineering (PROMISE) . ACM, 2022, pp. 62–71
2022
Cited alongside, same era.

Then

A. Sarkar, A. D. Gordon, C. Negreanu, C. Poelitz, S. S. Ragavan, and B. Zorn, “What is it like to program with artificial intelligence?” arXiv preprint abs/2208.06213 , 2022
Original
2022
Later among the works it cites.
H. Hata, N. Novielli, S. Baltes, R. G. Kula, and C. Treude, “Github discussions: An exploratory study of early adoption,” Empirical Software Engineering , vol. 27, no. 1, pp. 1–32, 2022
2022
Later among the works it cites.
C. Bird, D. Ford, T. Zimmermann, N. Forsgren, E. Kalliamvakou, T. Lowdermilk, and I. Gazit, “Taking flight with copilot: Early insights and opportunities of ai-powered pair-programming tools,” ACM Queue , vol. 20, no. 6, pp. 35––57, 2023
2023
Closest in time.
N. Al Madi, “How readable is model-generated code? examining readability and visual inspection of github copilot,” in Proc. of the 37th International Conference on Automated Software Engineering (ASE) . ACM, 2023, pp. 1–5
2023
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

A. M. Dakhel, V. Majdinasab, A. Nikanjam, F. Khomh, M. C. Desmarais, Z. Ming, and Jiang, “Github copilot ai pair programmer: Asset or liability?” arXiv preprint abs/2206.15331 , 2022

2022

Cited alongside, same era.

N. Nguyen and S. Nadi, “An empirical evaluation of github copilot’s code suggestions,” in Proc. of the 19th IEEE/ACM International Conference on Mining Software Repositories (MSR) . IEEE, 2022, pp. 1–5

2022

Cited alongside, same era.

GitHub Copilot · Your AI pair programmer , https://github.com/features/copilot

Cited in the paper.

GitHub Discussions: Copilot Category , https://github.com/community/community/discussions/categories/copilot

Cited in the paper.

C. Wang, J. Hu, C. Gao, Y. Jin, T. Xie, H. Huang, Z. Lei, and Y. Deng, “Practitioners’ expectations on code completion,” arXiv preprint abs/2301.03846 , 2023

Original

2023

Closest in time.

B. Zhang, P. Liang, X. Zhou, A. Ahmad, and M. Waseem, Dataset of the Paper: Practices and Challenges of Using GitHub Copilot: An Empirical Study , 2023, https://doi.org/10.5281/zenodo.7604508

2023

Closest in time.