2023

Is Stack Overflow Obsolete? An Empirical Study of the Characteristics of ChatGPT Answers to Stack Overflow Questions

Kabir, Samia, Udo-Imeh, David N., Kou, Bonan et al.

Understand

Q&A platforms have been crucial for the online help-seeking behavior of programmers.

However, the recent popularity of ChatGPT is altering this trend.
Despite this popularity, no comprehensive study has been conducted to evaluate the characteristics of ChatGPT's answers to programming questions.
To bridge the gap, we conducted the first in-depth analysis of ChatGPT answers to 517 programming questions on Stack Overflow and examined the correctness, consistency, comprehensiveness, and conciseness of ChatGPT answers.

Built on

Measuring nominal scale agreement among many raters
Joseph L Fleiss. 1971 · 1971
Earlier work this paper cites.
Software psychology: Human factors in computer and information systems (Winthrop computer systems series)
Ben Shneiderman. 1980 · 1980
Earlier work this paper cites.
An introduction to qualitative research
Beverley Hancock, Elizabeth Ockleford, and Kate Windridge. 2001 · 2001
Earlier work this paper cites.
Mixed methods research: A research paradigm whose time has come
R Burke Johnson and Anthony J Onwuegbuzie. 2004 · 2004
Earlier work this paper cites.
Using thematic analysis in psychology
Virginia Braun and Victoria Clarke. 2006 · 2006
Earlier work this paper cites.
Likert scales, levels of measurement and the “laws” of statistics
Geoff Norman. 2010 · 2010
Earlier work this paper cites.
The impact of social media on software engineering practices and tools. In Proceedings of the FSE/SDP workshop on Future of software engineering research . 359–364
Margaret-Anne Storey, Christoph Treude, Arie Van Deursen, and Li-Te Cheng. 2010 · 2010
Earlier work this paper cites.
Applied thematic analysis
Greg Guest, Kathleen M MacQueen, and Emily E Namey. 2011 · 2011
Earlier work this paper cites.
Design lessons from the fastest q&a site in the west. In Proceedings of the SIGCHI conference on Human factors in computing systems . 2857–2866
Lena Mamykina, Bella Manoim, Manas Mittal, George Hripcsak, and Björn Hartmann. 2011 · 2011
Earlier work this paper cites.
How do programmers ask and answer questions on the web?(nier track). In Proceedings of the 33rd international conference on software engineering . 804–807
Christoph Treude, Ohad Barzilay, and Margaret-Anne Storey. 2011 · 2011
Earlier work this paper cites.
Harnessing stack overflow for the ide. In 2012 Third International Workshop on Recommendation Systems for Software Engineering (RSSE) . IEEE, 26–30
Alberto Bacchelli, Luca Ponzanelli, and Michele Lanza. 2012 · 2012
Earlier work this paper cites.
The spread of emotion via Facebook. In Proceedings of the SIGCHI conference on human factors in computing systems . 767–770
Adam DI Kramer. 2012 · 2012
Earlier work this paper cites.
What makes a good code example?: A study of programming Q&A in StackOverflow. In 2012 28th IEEE International Conference on Software Maintenance (ICSM) . IEEE, 25–34
Seyed Mehdi Nasehi, Jonathan Sillito, Frank Maurer, and Chris Burns. 2012 · 2012
Earlier work this paper cites.
Why, when, and what: analyzing stack overflow questions by topic, type, and code. In 2013 10th Working conference on mining software repositories (MSR) . IEEE, 53–56
Miltiadis Allamanis and Charles Sutton. 2013 · 2013
Earlier work this paper cites.
Answering questions about unanswered questions of stack overflow. In 2013 10th Working Conference on Mining Software Repositories (MSR) . IEEE, 97–100
Muhammad Asaduzzaman, Ahmed Shah Mashiyat, Chanchal K Roy, and Kevin A Schneider. 2013 · 2013
Earlier work this paper cites.
On the personality traits of stackoverflow users. In 2013 IEEE international conference on software maintenance . IEEE, 460–463
Blerina Bazelli, Abram Hindle, and Eleni Stroulia. 2013 · 2013
Earlier work this paper cites.
Stackoverflow and github: Associations between software development and crowdsourced knowledge. In 2013 International Conference on Social Computing . IEEE, 188–195
Bogdan Vasilescu, Vladimir Filkov, and Alexander Serebrenik. 2013 · 2013
Earlier work this paper cites.
Ranking crowd knowledge to assist software development. In Proceedings of the 22nd International Conference on Program Comprehension . 72–82
Lucas BL De Souza, Eduardo C Campos, and Marcelo de A Maia. 2014 · 2014
Earlier work this paper cites.
Sequence labeling with multiple annotators
Filipe Rodrigues, Francisco Pereira, and Bernardete Ribeiro. 2014 · 2014
Earlier work this paper cites.
Mining successful answers in stack overflow. In 2015 IEEE/ACM 12th Working Conference on Mining Software Repositories . IEEE, 430–433
Fabio Calefato, Filippo Lanubile, Maria Concetta Marasciulo, and Nicole Novielli. 2015 · 2015
Earlier work this paper cites.
The development and psychometric properties of LIWC2015
James W Pennebaker, Ryan L Boyd, Kayla Jordan, and Kate Blackburn. 2015 · 2015
Earlier work this paper cites.
What do developers use the crowd for? a study using stack overflow
Rabe Abdalkareem, Emad Shihab, and Juergen Rilling. 2017 · 2017
Earlier work this paper cites.
Truth of varying shades: Analyzing language in fake news and political fact-checking. In Proceedings of the 2017 conference on empirical methods in natural language processing . 2931–2937
Hannah Rashkin, Eunsol Choi, Jin Yea Jang, Svitlana Volkova, and Yejin Choi. 2017 · 2017
Earlier work this paper cites.
What do developers search for on the web?
Xin Xia, Lingfeng Bao, David Lo, Pavneet Singh Kochhar, Ahmed E Hassan, and Zhenchang Xing. 2017 · 2017
Earlier work this paper cites.
Deeppath: A reinforcement learning method for knowledge graph reasoning
Original
Wenhan Xiong, Thien Hoang, and William Yang Wang. 2017 · 2017
Earlier work this paper cites.
’It’s Reducing a Human Being to a Percentage’ Perceptions of Justice in Algorithmic Decisions. In Proceedings of the 2018 Chi conference on human factors in computing systems . 1–14
Reuben Binns, Max Van Kleek, Michael Veale, Ulrik Lyngs, Jun Zhao, and Nigel Shadbolt. 2018 · 2018
Earlier work this paper cites.
How to ask for technical help? Evidence-based guidelines for writing questions on Stack Overflow
Fabio Calefato, Filippo Lanubile, and Nicole Novielli. 2018a · 2018
Earlier work this paper cites.
Linguistic signals under misinformation and fact-checking: Evidence from user comments on social media
Shan Jiang and Christo Wilson. 2018 · 2018
Earlier work this paper cites.
Evaluating how developers use general-purpose web-search for code retrieval. In Proceedings of the 15th International Conference on Mining Software Repositories . 465–475
Md Masudur Rahman, Jed Barson, Sydney Paul, Joshua Kayani, Federico Andrés Lois, Sebastián Fernandez Quezada, Christopher Parnin, Kathryn T Stolee, and Baishakhi Ray. 2018 · 2018
Earlier work this paper cites.

Similar

Assessing the factual accuracy of generated text. In proceedings of the 25th ACM SIGKDD international conference on knowledge discovery & data mining . 166–175
Ben Goodrich, Vinay Rao, Peter J Liu, and Mohammad Saleh. 2019 · 2019
Cited alongside, same era.
The signals that potential contributors look for when choosing open-source projects
Huilian Sophie Qiu, Yucen Lily Li, Susmita Padala, Anita Sarma, and Bogdan Vasilescu. 2019 · 2019
Cited alongside, same era.
The emergence of deepfake technology: A review
Mika Westerlund. 2019 · 2019
Cited alongside, same era.
Quality estimation & interpretability for code translation
Original
Mayank Agarwal, Kartik Talamadupula, Stephanie Houde, Fernando Martinez, Michael Muller, John Richards, Steven Ross, and Justin D Weisz. 2020 · 2020
Cited alongside, same era.
Climbing towards NLU: On meaning, form, and understanding in the age of data. In Proceedings of the 58th annual meeting of the association for computational linguistics . 5185–5198
Emily M Bender and Alexander Koller. 2020 · 2020
Cited alongside, same era.
We Need to Talk About Sentiment Analysis
Kristof Boghe. 2020 · 2020
Cited alongside, same era.
A review: Knowledge reasoning over knowledge graph
Xiaojun Chen, Shengbin Jia, and Yang Xiang. 2020 · 2020
Cited alongside, same era.
AI-Generated Text Is the Scariest Deepfake of All
Renee DiResta. 2020 · 2020
Cited alongside, same era.
Realtoxicityprompts: Evaluating neural toxic degeneration in language models
Original
Samuel Gehman, Suchin Gururangan, Maarten Sap, Yejin Choi, and Noah A Smith. 2020 · 2020
Cited alongside, same era.
COVID-19–related infodemic and its impact on public health: A global social media analysis
Md Saiful Islam, Tonmoy Sarkar, Sazzad Hossain Khan, Abu-Hena Mostofa Kamal, SM Murshid Hasan, Alamgir Kabir, Dalia Yeasmin, Mohammad Ariful Islam, Kamal Ibne Amin Chowdhury, Kazi Selim Anwar, et al · 2020
Cited alongside, same era.
On faithfulness and factuality in abstractive summarization
Original
Joshua Maynez, Shashi Narayan, Bernd Bohnet, and Ryan McDonald. 2020 · 2020
Cited alongside, same era.
Analyzing web search behavior for software engineering tasks. In 2020 IEEE International Conference on Big Data (Big Data) . IEEE, 768–777
Nikitha Rao, Chetan Bansal, Thomas Zimmermann, Ahmed Hassan Awadallah, and Nachiappan Nagappan. 2020 · 2020
Cited alongside, same era.

Then

Is github copilot a substitute for human pair-programming? an empirical study. In Proceedings of the ACM/IEEE 44th International Conference on Software Engineering: Companion Proceedings . 319–321
Saki Imai. 2022 · 2022
Later among the works it cites.
SOSum: a dataset of stack overflow post summaries. In Proceedings of the 19th International Conference on Mining Software Repositories . 247–251
Bonan Kou, Yifeng Di, Muhao Chen, and Tianyi Zhang. 2022 · 2022
Later among the works it cites.
All the news that’s fit to fabricate: AI-generated text as a tool of media misinformation
Sarah Kreps, R Miles McCain, and Miles Brundage. 2022 · 2022
Later among the works it cites.
"Did you miss my comment or what?" understanding toxicity in open source discussions. In Proceedings of the 44th International Conference on Software Engineering . 710–722
Courtney Miller, Sophie Cohen, Daniel Klug, Bogdan Vasilescu, and Christian Kastner. 2022 · 2022
Later among the works it cites.
Interactive and visual prompt engineering for ad-hoc task adaptation with large language models
Hendrik Strobelt, Albert Webson, Victor Sanh, Benjamin Hoover, Johanna Beyer, Hanspeter Pfister, and Alexander M Rush. 2022 · 2022
Later among the works it cites.
Expectation vs. experience: Evaluating the usability of code generation tools powered by large language models. In Chi conference on human factors in computing systems extended abstracts . 1–7
Priyan Vaithilingam, Tianyi Zhang, and Elena L Glassman. 2022 · 2022
Later among the works it cites.
Large language models are human-level prompt engineers
Original
Yongchao Zhou, Andrei Ioan Muresanu, Ziwen Han, Keiran Paster, Silviu Pitis, Harris Chan, and Jimmy Ba. 2022 · 2022
Later among the works it cites.
The internal state of an llm knows when its lying
Original
Amos Azaria and Tom Mitchell. 2023 · 2023
Closest in time.
Grounded copilot: How programmers interact with code-generating models
Shraddha Barke, Michael B James, and Nadia Polikarpova. 2023 · 2023
Closest in time.
A categorical archive of chatgpt failures
Original
Ali Borji. 2023 · 2023
Closest in time.
Github copilot ai pair programmer: Asset or liability?
Arghavan Moradi Dakhel, Vahid Majdinasab, Amin Nikanjam, Foutse Khomh, Michel C Desmarais, and Zhen Ming Jack Jiang. 2023 · 2023
Closest in time.
Stack Exchange Data Dump
Stack Exchange. 2023 · 2023
Closest in time.
GitHub Copilot X: The AI-powered developer experience
GitHub. 2023a · 2023
Closest in time.
How close is chatgpt to human experts? comparison corpus, evaluation, and detection
Original
Biyang Guo, Xin Zhang, Ziyuan Wang, Minqi Jiang, Jinran Nie, Yuxuan Ding, Jianwei Yue, and Yupeng Wu. 2023 · 2023
Closest in time.
ChatGPT: Jack of all trades, master of none
Jan Kocoń, Igor Cichecki, Oliwier Kaszyca, Mateusz Kochanek, Dominika Szydło, Joanna Baran, Julita Bielaniewicz, Marcin Gruza, Arkadiusz Janz, Kamil Kanclerz, et al · 2023
Closest in time.
Automated Summarization of Stack Overflow Posts. In 45th IEEE/ACM International Conference on Software Engineering, ICSE 2023, Melbourne, Australia, May 14-20, 2023 . IEEE, 1853–1865
Bonan Kou, Muhao Chen, and Tianyi Zhang. 2023 · 2023
Closest in time.
Chatgpt or human? detect and explain. explaining decisions of machine learning model for detecting short chatgpt-generated text
Original
Sandra Mitrović, Davide Andreoletti, and Omran Ayoub. 2023 · 2023
Closest in time.
ChatGPT: Optimizing Language Models for Dialogue
OpenAI. 2023 · 2023
Closest in time.
Insights into Stack Overflow’s traffic
Stack Overflow. 2023b · 2023
Closest in time.
Check your facts and try again: Improving large language models with external knowledge and automated feedback
Original
Baolin Peng, Michel Galley, Pengcheng He, Hao Cheng, Yujia Xie, Yu Hu, Qiuyuan Huang, Lars Liden, Zhou Yu, Weizhu Chen, et al · 2023
Closest in time.
Will ChatGPT replace Stack Overflow?
Quora. 2023 · 2023
Closest in time.
The programmer’s assistant: Conversational interaction with a large language model for software development. In Proceedings of the 28th International Conference on Intelligent User Interfaces . 491–514
Steven I Ross, Fernando Martinez, Stephanie Houde, Michael Muller, and Justin D Weisz. 2023 · 2023
Closest in time.
Analysis of Novices’ Web-Based Help-Seeking Behavior While Programming. In Proceedings of the 54th ACM Technical Symposium on Computer Science Education V. 1 . 945–951
James Skripchuk, Neil Bennett, Jeffrey Zhang, Eric Li, and Thomas Price. 2023 · 2023
Closest in time.
Human-Centered Responsible Artificial Intelligence: Current & Future Trends. In Extended Abstracts of the 2023 CHI Conference on Human Factors in Computing Systems . 1–4
Mohammad Tahaei, Marios Constantinides, Daniele Quercia, Sean Kennedy, Michael Muller, Simone Stumpf, Q Vera Liao, Ricardo Baeza-Yates, Lora Aroyo, Jess Holbrook, et al · 2023
Closest in time.
GitHub Copilot vs. ChatGPT: Which Tool Is Better for Software Development?
Christopher Tozzi. 2023 · 2023
Closest in time.
Generation probabilities are not enough: Exploring the effectiveness of uncertainty highlighting in AI-powered code completions
Original
Helena Vasconcelos, Gagan Bansal, Adam Fourney, Q Vera Liao, and Jennifer Wortman Vaughan. 2023 · 2023
Closest in time.
Synthetic lies: Understanding ai-generated misinformation and evaluating algorithmic and human solutions. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems . 1–20
Jiawei Zhou, Yixuan Zhang, Qianni Luo, Andrea G Parker, and Munmun De Choudhury. 2023 · 2023
Closest in time.
Summarizing source code using a neural attention model. In 54th Annual Meeting of the Association for Computational Linguistics 2016 . Association for Computational Linguistics, 2073–2083
Srinivasan Iyer, Ioannis Konstas, Alvin Cheung, and Luke Zettlemoyer. 2016 · 2083
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…