2024

CodePori: Large-Scale System for Autonomous Software Development Using Multi-Agent Technology

Rasheed, Zeeshan, Waseem, Muhammad, Kemell, Kai-Kristian et al.

Understand

Context: LLM-based multi-agent systems enable automation and decision support in software development, yet existing studies rely on benchmark datasets offering only binary pass-or-fail results, limiting insight into real-world applicability.

Objective: This study empirically investigates the potential and limitations of LLM-based agents in autonomous software development tasks.
Method: A two-phase approach was employed: developing a multi-agent system, CodePori, for automated code generation, and conducting participant-based evaluation to assess practical performance.
Results: Participant feedback reveals key strengths, challenges, and areas for improvement in LLM-based multi-agent systems, highlighting aspects missed by standard code-generation benchmarks.

Built on

T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, et al., Language models are few-shot learners, Advances in neural information processing systems 33 (2020) 1877–1901
1901
Earlier work this paper cites.
Z. Feng, D. Guo, D. Tang, N. Duan, X. Feng, M. Gong, L. Shou, B. Qin, T. Liu, D. Jiang, et al., Codebert: A pre-trained model for programming and natural languages, arXiv preprint arXiv:2002.08155
Original
2002
Earlier work this paper cites.
M. Tufano, D. Drain, A. Svyatkovskiy, S. K. Deng, N. Sundaresan, Unit test case generation with transformers and focal context, arXiv preprint arXiv:2009.05617
Original
2009
Earlier work this paper cites.
P. Runeson, M. Höst, Guidelines for conducting and reporting case study research in software engineering, Empirical software engineering 14 (2009) 131–164
2009
Earlier work this paper cites.
X. Gu, H. Zhang, S. Kim, Deep code search, in: Proceedings of the 40th International Conference on Software Engineering, 2018, pp. 933–944
2018
Earlier work this paper cites.
A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, I. Sutskever, et al., Language models are unsupervised multitask learners, OpenAI blog 1 (8) (2019) 9
2019
Earlier work this paper cites.
L. Floridi, M. Chiriatti, Gpt-3: Its nature, scope, limits, and consequences, Minds and Machines 30 (2020) 681–694
2020
Earlier work this paper cites.
B. Wang, A. Komatsuzaki, Gpt-j-6b: A 6 billion parameter autoregressive language model (2021)
2021
Earlier work this paper cites.
L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray, et al., Training language models to follow instructions with human feedback, Advances in Neural Information Processing Systems 35 (2022) 27730–27744
2022
Earlier work this paper cites.

Similar

Y. Li, D. Choi, J. Chung, N. Kushman, J. Schrittwieser, R. Leblond, T. Eccles, J. Keeling, F. Gimeno, A. Dal Lago, et al., Competition-level code generation with alphacode, Science 378 (6624) (2022) 1092–1097
2022
Cited alongside, same era.
D. Rothman, A. Gulli, Transformers for Natural Language Processing: Build, train, and fine-tune deep neural network architectures for NLP with Python, PyTorch, TensorFlow, BERT, and GPT-3, Packt Publishing Ltd, 2022
2022
Cited alongside, same era.
L. Tunstall, L. Von Werra, T. Wolf, Natural language processing with transformers, " O’Reilly Media, Inc.", 2022
2022
Cited alongside, same era.
F. F. Xu, U. Alon, G. Neubig, V. J. Hellendoorn, A systematic evaluation of large language models of code, in: Proceedings of the 6th ACM SIGPLAN International Symposium on Machine Programming, 2022, pp. 1–10
2022
Cited alongside, same era.
S. Arora, A. Narayan, M. F. Chen, L. Orr, N. Guha, K. Bhatia, I. Chami, C. Re, Ask me anything: A simple strategy for prompting language models, in: The Eleventh International Conference on Learning Representations, 2022
2022
Cited alongside, same era.
C. Treude, Navigating complexity in software engineering: A prototype for comparing gpt-n solutions, in: 2023 IEEE/ACM 5th International Workshop on Bots in Software Engineering (BotSE), IEEE, 2023, pp. 1–5
2023
Cited alongside, same era.
L. Belzner, T. Gabor, M. Wirsing, Large language model assisted software engineering: prospects, challenges, and a case study, in: International Conference on Bridging the Gap between AI and Reality, Springer, 2023, pp. 355–374
2023
Cited alongside, same era.
X. Hu, H. K. Dam, Future of software engineering@ icse 2023, in: 2023 IEEE/ACM International Conference on Software Engineering: Future of Software Engineering (ICSE-FoSE), IEEE, 2023, pp. 1–3
2023
Cited alongside, same era.
Q. Gu, Llm-based code generation method for golang compiler testing, in: Proceedings of the 31st ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering, 2023, pp. 2201–2203
2023
Cited alongside, same era.
A. Radford, K. Narasimhan, T. Salimans, I. Sutskever, et al., Improving language understanding by generative pre-training
Cited in the paper.
M. Allamanis, M. Brockschmidt, M. Khademi, Learning to represent programs with graphs, arXiv preprint arXiv:1711.00740
Original
Cited in the paper.
J. Wei, Y. Tay, R. Bommasani, C. Raffel, B. Zoph, S. Borgeaud, D. Yogatama, M. Bosma, D. Zhou, D. Metzler, et al., Emergent abilities of large language models, arXiv preprint arXiv:2206.07682
Original
Cited in the paper.

Then

A. Chowdhery, S. Narang, J. Devlin, M. Bosma, G. Mishra, A. Roberts, P. Barham, H. W. Chung, C. Sutton, S. Gehrmann, et al., Palm: Scaling language modeling with pathways, Journal of Machine Learning Research 24 (240) (2023) 1–113
2023
Later among the works it cites.
P. Hacker, A. Engel, M. Mauer, Regulating chatgpt and other large generative ai models, in: Proceedings of the 2023 ACM Conference on Fairness, Accountability, and Transparency, 2023, pp. 1112–1123
2023
Later among the works it cites.
Y. Feng, S. Vanam, M. Cherukupally, W. Zheng, M. Qiu, H. Chen, Investigating code generation performance of chat-gpt with crowdsourcing social data, in: Proceedings of the 47th IEEE Computer Software and Applications Conference, 2023, pp. 1–10
2023
Later among the works it cites.
A. Hörnemalm, Chatgpt as a software development tool: The future of development (2023)
2023
Later among the works it cites.
M. Roberts, H. Thakur, C. Herlihy, C. White, S. Dooley, To the cutoff… and beyond? a longitudinal perspective on llm data contamination, in: The Twelfth International Conference on Learning Representations, 2023
2023
Later among the works it cites.
F. Urrutia, R. Araya, Who’s the best detective? large language models vs. traditional machine learning in detecting incoherent fourth grade math answers, Journal of Educational Computing Research 61 (8) (2024) 187–218
2024
Closest in time.
Z. Rasheed, Dataset of the Paper “CodePori: Large Scale System for Autonomous Software Development by Using Multi-Agents”, https://doi.org/10.5281/zenodo.13755415 (2024)
2024
Closest in time.
Z. Rasheed, M. A. Sami, P. Abrahamsson, Codepori , accessed: 2024-09-12 (2024). URL https://github.com/GPT-Laboratory/CodePori
2024
Closest in time.
F. Quin, D. Weyns, M. Galster, C. C. Silva, A/b testing: a systematic literature review, Journal of Systems and Software (2024) 112011
2024
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…