Can LLMs Replace Manual Annotation of Software Engineering Artifacts?

J. Cohen, “A coefficient of agreement for nominal scales,” Educational and psychological measurement , vol. 20, no. 1, pp. 37–46, 1960

1960

Earlier work this paper cites.

J. M. Morse, M. Barrett, M. Mayan, K. Olson, and J. Spiers, “Verification strategies for establishing reliability and validity in qualitative research,” International journal of qualitative methods , vol. 1, no. 2, pp. 13–22, 2002

2002

Earlier work this paper cites.

M. Lombard, J. Snyder-Duch, and C. C. Bracken, “Content analysis in mass communication: Assessment and reporting of intercoder reliability,” Human communication research , vol. 28, no. 4, pp. 587–604, 2002

2002

Earlier work this paper cites.

L. Finlay, ““outing” the researcher: The provenance, process, and practice of reflexivity,” Qualitative health research , vol. 12, no. 4, pp. 531–545, 2002

2002

Earlier work this paper cites.

S. Easterbrook, J. Singer, M.-A. Storey, and D. Damian, “Selecting empirical methods for software engineering research,” Guide to advanced empirical software engineering , pp. 285–311, 2008

2008

Earlier work this paper cites.

G. Sridhara, E. Hill, D. Muppaneni, L. Pollock, and K. Vijay-Shanker, “Towards automatically generating summary comments for java methods,” in Proceedings of the 25th IEEE/ACM international conference on Automated software engineering , 2010, pp. 43–52

2010

Earlier work this paper cites.

K. Krippendorff, Content analysis: An introduction to its methodology . Sage publications, 2018

2018

Earlier work this paper cites.

V. J. Hellendoorn, C. Bird, E. T. Barr, and M. Allamanis, “Deep learning type inference,” in Proceedings of the 2018 ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering, ESEC/SIGSOFT , 2018, pp. 152–162. [Online]. Available: https://doi.org/10.1145/3236024.3236051

2018

Earlier work this paper cites.

Y. Zhu and M. Pan, “Automatic code summarization: A systematic literature review,” arXiv preprint arXiv:1909.04352 , 2019

Original

2019

Earlier work this paper cites.

R. Futrell, E. Wilcox, T. Morita, P. Qian, M. Ballesteros, and R. Levy, “Neural language models as psycholinguistic subjects: Representations of syntactic state,” arXiv preprint arXiv:1903.03260 , 2019

Original

2019

Earlier work this paper cites.

A. LeClair, S. Jiang, and C. McMillan, “A neural model for generating natural language summaries of program subroutines,” in 2019 IEEE/ACM 41st International Conference on Software Engineering (ICSE) . IEEE, 2019, pp. 795–806

2019

Earlier work this paper cites.

M. Kamp, P. Kreutzer, and M. Philippsen, “Sesame: A data set of semantically similar java methods,” in 2019 IEEE/ACM 16th International Conference on Mining Software Repositories (MSR) . IEEE, 2019, pp. 529–533

2019

Earlier work this paper cites.

Z. Chen, S. Kommrusch, M. Tufano, L. Pouchet, D. Poshyvanyk, and M. Monperrus, “SequenceR: Sequence-to-sequence learning for end-to-end program repair,” IEEE Trans. Software Eng. , vol. 47, no. 9, pp. 1943–1959, 2021. [Online]. Available: https://doi.org/10.1109/TSE.2019.2940179

2019

Earlier work this paper cites.

R. S. Malik, J. Patra, and M. Pradel, “NL2Type: Inferring JavaScript function types from natural language information,” in Proceedings of the 41st International Conference on Software Engineering, ICSE 2019, Montreal, QC, Canada, May 25-31, 2019 , 2019, pp. 304–315. [Online]. Available: https://doi.org/10.1109/ICSE.2019.00045

2019

Earlier work this paper cites.

T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell et al. , “Language models are few-shot learners,” Advances in neural information processing systems , vol. 33, pp. 1877–1901, 2020

2020

Earlier work this paper cites.

M. Pradel, G. Gousios, J. Liu, and S. Chandra, “Typewriter: Neural type prediction with search-based validation,” in ESEC/FSE ’20: 28th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering, Virtual Event, USA, November 8-13, 2020 , 2020, pp. 209–220. [Online]. Available: https://doi.org/10.1145/3368089.3409715

2020

Earlier work this paper cites.

M. Allamanis, E. T. Barr, S. Ducousso, and Z. Gao, “Typilus: neural type hints,” in Proceedings of the 41st ACM SIGPLAN International Conference on Programming Language Design and Implementation, PLDI , 2020, pp. 91–105. [Online]. Available: https://doi.org/10.1145/3385412.3385997

2020

Earlier work this paper cites.

D. Roy, S. Fakhoury, and V. Arnaoudova, “Reassessing automatic evaluation metrics for code summarization tasks,” in Proceedings of the 29th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering , 2021, pp. 1105–1116

2021

Earlier work this paper cites.

J. Fischbach, J. Frattini, A. Spaans, M. Kummeth, A. Vogelsang, D. Mendez, and M. Unterkalmsteiner, “Automatic detection of causality in requirement artifacts: the cira approach,” in Requirements Engineering: Foundation for Software Quality: 27th International Working Conference, REFSQ 2021, Essen, Germany, April 12–15, 2021, Proceedings 27 . Springer, 2021, pp. 19–36

2021

Earlier work this paper cites.

M. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. de Oliveira Pinto, J. Kaplan, H. Edwards, Y. Burda, N. Joseph, G. Brockman, A. Ray, R. Puri, G. Krueger, M. Petrov, H. Khlaaf, G. Sastry, P. Mishkin, B. Chan, S. Gray, N. Ryder, M. Pavlov, A. Power, L. Kaiser, M. Bavarian, C. Winter, P. Tillet, F. P. Such, D. Cummings, M. Plappert, F. Chantzis, E. Barnes, A. Herbert-Voss, W. H. Guss, A. Nichol, A. Paino, N. Tezak, J. Tang, I. Babuschkin, S. Balaji, S. Jain, W. Saunders, C. Hesse, A. N. Carr, J. Leike, J. Achiam, V. Misra, E. Morikawa, A. Radford, M. Knight, M. Brundage, M. Murati, K. Mayer, P. Welinder, B. McGrew, D. Amodei, S. McCandlish, I. Sutskever, and W. Zaremba, “Evaluating large language models trained on code,” CoRR , vol. abs/2107.03374, 2021. [Online]. Available: https://arxiv.org/abs/2107.03374

Original

2021

Earlier work this paper cites.

C. Zhang, J. Wang, Q. Zhou, T. Xu, K. Tang, H. Gui, and F. Liu, “A survey of automatic source code summarization,” Symmetry , vol. 14, no. 3, p. 471, 2022

2022

Earlier work this paper cites.

H. J. Kang, K. L. Aw, and D. Lo, “Detecting false alarms from automatic static analysis tools: How far are we?” in Proceedings of the 44th International Conference on Software Engineering , 2022, pp. 698–709

2022

Earlier work this paper cites.

S. Haque, Z. Eberhart, A. Bansal, and C. McMillan, “Semantic similarity metrics for evaluating source code summarization,” in Proceedings of the 30th IEEE/ACM International Conference on Program Comprehension , 2022, pp. 36–47

2022

Cited alongside, same era.

T. Ahmed and P. Devanbu, “Few-shot training llms for project-specific code-summarization,” in Proceedings of the 37th IEEE/ACM International Conference on Automated Software Engineering , 2022, pp. 1–5

2022

Cited alongside, same era.

J. Patra and M. Pradel, “Nalin: learning from runtime behavior to find name-value inconsistencies in jupyter notebooks,” in Proceedings of the 44th International Conference on Software Engineering , 2022, pp. 1469–1481

2022

Cited alongside, same era.

X. Hu, X. Xia, D. Lo, Z. Wan, Q. Chen, and T. Zimmermann, “Practitioners’ expectations on automated code comment generation,” in Proceedings of the 44th International Conference on Software Engineering , 2022, pp. 1693–1705

2022

M. Reid, N. Savinov, D. Teplyashin, D. Lepikhin, T. Lillicrap, J.-b. Alayrac, R. Soricut, A. Lazaridou, O. Firat, J. Schrittwieser et al. , “Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context,” arXiv preprint arXiv:2403.05530 , 2024

Original

2024

Closest in time.

A. Bavaresco, R. Bernardi, L. Bertolazzi, D. Elliott, R. Fernández, A. Gatt, E. Ghaleb, M. Giulianelli, M. Hanna, A. Koller et al. , “Llms instead of human judges? a large scale empirical study across 20 nlp evaluation tasks,” arXiv preprint arXiv:2406.18403 , 2024

Original

2024

Closest in time.

O. Dunay, D. Cheng, A. Tait, P. Thakkar, P. C. Rigby, A. Chiu, I. Ahmad, A. Ganesan, C. Maddila, V. Murali et al. , “Multi-line ai-assisted code authoring,” in Companion Proceedings of the 32nd ACM International Conference on the Foundations of Software Engineering , 2024, pp. 150–160

2024

Closest in time.

A. Eghbali and M. Pradel, “De-hallucinator: Iterative grounding for llm-based code completion,” CoRR , vol. abs/2401.01701, 2024. [Online]. Available: https://doi.org/10.48550/arXiv.2401.01701

Original