Fetching the paper…

Reading the bibliography…

2023

Humans are Still Better than ChatGPT: Case of the IEEEXtreme Competition

Koubaa, Anis, Qureshi, Basit, Ammar, Adel et al.

Original
PDF
Post on X

Understand

Since the release of ChatGPT, numerous studies have highlighted the remarkable performance of ChatGPT, which often rivals or even surpasses human capabilities in various tasks and domains.

However, this paper presents a contrasting perspective by demonstrating an instance where human performance excels in typical tasks suited for ChatGPT, specifically in the domain of computer programming.
We utilize the IEEExtreme Challenge competition as a benchmark, a prestigious, annual international programming contest encompassing a wide range of problems with different complexities.
To conduct a thorough evaluation, we selected and executed a diverse set of 102 challenges, drawn from five distinct IEEExtreme editions, using three major programming languages: Python, Java, and C++.

Humans are Still Better than ChatGPT: Case of the IEEEXtreme Competition · Around

Built on

Y. Liu, M. Ott, N. Goyal, J. Du, M. Joshi, D. Chen, O. Levy, M. Lewis, L. Zettlemoyer, and V. Stoyanov, “Roberta: A robustly optimized bert pretraining approach,” arXiv preprint arXiv:1907.11692 , 2019
Original
2019
Earlier work this paper cites.
Y. Hu, U. Z. Ahmed, S. Mechtaev, B. Leong, and A. Roychoudhury, “Re-factoring based program repair applied to programming assignments,” in 2019 34th IEEE/ACM International Conference on Automated Software Engineering (ASE) . IEEE, 2019, pp. 388–398
2019
Earlier work this paper cites.
W. X. Zhao, K. Zhou, J. Li, T. Tang, X. Wang, Y. Hou, Y. Min, B. Zhang, J. Zhang, Z. Dong, Y. Du, C. Yang, Y. Chen, Z. Chen, J. Jiang, R. Ren, Y. Li, X. Tang, Z. Liu, P. Liu, J.-Y. Nie, and J.-R. Wen, “A survey of large language models,” 2023
2023
Earlier work this paper cites.
J. Qiu, L. Li, J. Sun, J. Peng, P. Shi, R. Zhang, Y. Dong, K. Lam, F. P. W. Lo, B. Xiao, W. Yuan, D. Xu, and B. Lo, “Large ai models in health informatics: Applications, challenges, and the future,” 2023
2023
Earlier work this paper cites.
S. Wu, O. Irsoy, S. Lu, V. Dabravolski, M. Dredze, S. Gehrmann, P. Kambadur, D. Rosenberg, and G. Mann, “Bloomberggpt: A large language model for finance,” 2023
2023
Earlier work this paper cites.
A. Koubaa, W. Boulila, L. Ghouti, A. Alzahem, and S. Latif, “Exploring ChatGPT capabilities and limitations: A critical review of the nlp game changer,” 2023
2023
Earlier work this paper cites.
B. Guo, X. Zhang, Z. Wang, M. Jiang, J. Nie, Y. Ding, J. Yue, and Y. Wu, “How close is chatgpt to human experts? comparison corpus, evaluation, and detection,” 2023
2023
Earlier work this paper cites.
Y. Bang, S. Cahyawijaya, N. Lee, W. Dai, D. Su, B. Wilie, H. Lovenia, Z. Ji, T. Yu, W. Chung et al. , “A multitask, multilingual, multimodal evaluation of chatgpt on reasoning, hallucination, and interactivity,” arXiv preprint arXiv:2302.04023 , 2023
Original
2023
Earlier work this paper cites.
M. Ortega-Martín, Ó. García-Sierra, A. Ardoiz, J. Álvarez, J. C. Armenteros, and A. Alonso, “Linguistic ambiguity analysis in chatgpt,” arXiv preprint arXiv:2302.06426 , 2023
Original
2023
Earlier work this paper cites.
S. Liu, A. P. Wright, B. L. Patterson, J. P. Wanderer, R. W. Turer, S. D. Nelson, A. B. McCoy, D. F. Sittig, and A. Wright, “Assessing the value of chatgpt for clinical decision support optimization,” medRxiv , pp. 2023–02, 2023
2023
Earlier work this paper cites.
M. Sallam, “Chatgpt utility in healthcare education, research, and practice: Systematic review on the promising perspectives and valid concerns,” in Healthcare , vol. 11, no. 6. MDPI, 2023, p. 887
2023
Earlier work this paper cites.
L. Iftikhar et al. , “Docgpt: Impact of chatgpt-3 on health services as a virtual doctor,” EC Paediatrics , vol. 12, no. 1, pp. 45–55, 2023
2023
Earlier work this paper cites.

Similar

P. Hacker, A. Engel, and M. Mauer, “Regulating chatgpt and other large generative ai models,” arXiv preprint arXiv:2302.02337 , 2023
Original
2023
Cited alongside, same era.
S. Fergus, M. Botha, and M. Ostovar, “Evaluating academic answers generated using chatgpt,” Journal of Chemical Education , 2023
2023
Cited alongside, same era.
D. R. Cotton, P. A. Cotton, and J. R. Shipway, “Chatting and cheating: Ensuring academic integrity in the era of chatgpt,” Innovations in Education and Teaching International , pp. 1–12, 2023
2023
Cited alongside, same era.
S. Frieder, L. Pinchetti, R.-R. Griffiths, T. Salvatori, T. Lukasiewicz, P. C. Petersen, A. Chevalier, and J. Berner, “Mathematical capabilities of chatgpt,” arXiv preprint arXiv:2301.13867 , 2023
Original
2023
Cited alongside, same era.

Then

H. Tian, W. Lu, T. O. Li, X. Tang, S.-C. Cheung, J. Klein, and T. F. Bissyandé, “Is chatgpt the ultimate programming assistant–how far is it?” arXiv preprint arXiv:2304.11938 , 2023
Original
2023
Closest in time.
S. Biswas, “Role of chatgpt in computer programming.: Chatgpt in computer programming.” Mesopotamian Journal of Computer Science , vol. 2023, pp. 8–16, 2023
2023
Closest in time.
L. Avila-Chauvet, D. Mejía, and C. O. Acosta Quiroz, “Chatgpt as a support tool for online behavioral task programming,” Available at SSRN 4329020 , 2023
2023
Closest in time.
“Ieeextreme competition website,” 2023. [Online]. Available: https://ieeextreme.org/
2023
Closest in time.
“The international collegiate programming contest,” 2023. [Online]. Available: https://icpc.global/
2023
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

A. Tlili, B. Shehata, M. A. Adarkwah, A. Bozkurt, D. T. Hickey, R. Huang, and B. Agyemang, “What if the devil is my guardian angel: Chatgpt as a case study of using chatbots in education,” Smart Learning Environments , vol. 10, no. 1, p. 15, 2023

2023

Cited alongside, same era.

B. Qureshi, “Exploring the use of chatgpt as a tool for learning and assessment in undergraduate computer science curriculum: Opportunities and challenges,” arXiv preprint arXiv:2304.11214 , 2023

2023

Cited alongside, same era.

S. A. Prieto, E. T. Mengiste, and B. García de Soto, “Investigating the use of chatgpt for the scheduling of construction projects,” Buildings , vol. 13, no. 4, p. 857, 2023

2023

Cited alongside, same era.

M. Dowling and B. Lucey, “Chatgpt for (finance) research: The bananarama conjecture,” Finance Research Letters , vol. 53, p. 103662, 2023

2023

Cited alongside, same era.

A. B. Kocaballi, “Conversational ai-powered design: Chatgpt as designer, user, and product,” arXiv preprint arXiv:2302.07406 , 2023

2023

Cited alongside, same era.

C. Qin, A. Zhang, Z. Zhang, J. Chen, M. Yasunaga, and D. Yang, “Is chatgpt a general-purpose natural language processing task solver?” arXiv preprint arXiv:2302.06476 , 2023

2023

Cited alongside, same era.

A. Kashefi and T. Mukerji, “Chatgpt for programming numerical methods,” arXiv preprint arXiv:2303.12093 , 2023

2023

Cited alongside, same era.

H. Liu, R. Ning, Z. Teng, J. Liu, Q. Zhou, and Y. Zhang, “Evaluating the logical reasoning ability of chatgpt and gpt-4,” arXiv preprint arXiv:2304.03439 , 2023

2023

Cited alongside, same era.

“Google code jam,” 2023. [Online]. Available: https://codingcompetitions.withgoogle.com/codejam

2023

Closest in time.

“Meta hackercup,” 2023. [Online]. Available: https://www.facebook.com/codingcompetitions/hacker-cup

2023

Closest in time.

“International olympiad in informatics (ioi),” 2023. [Online]. Available: https://ioinformatics.org/

2023

Closest in time.

“Cs academy platform,” 2023. [Online]. Available: https://csacademy.com/ieeextreme

2023

Closest in time.

“Hackerrank platform,” 2023. [Online]. Available: https://www.hackerrank.com/ieeextreme7

2023

Closest in time.

“Hackerrank practice community website for ieeextreme competition,” 2023. [Online]. Available: https://www.hackerrank.com/contests/ieeextreme-challenges/challenges

2023

Closest in time.