2023

Revisiting Sentiment Analysis for Software Engineering in the Era of Large Language Models

Zhang, Ting, Irsan, Ivana Clairine, Thung, Ferdian et al.

Understand

Software development involves collaborative interactions where stakeholders express opinions across various platforms.

Recognizing the sentiments conveyed in these interactions is crucial for the effective development and ongoing maintenance of software systems.
For software products, analyzing the sentiment of user feedback, e.g., reviews, comments, and forum posts can provide valuable insights into user satisfaction and areas for improvement.
This can guide the development of future updates and features.

Built on

Language models are few-shot learners
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al · 1901
Earlier work this paper cites.
Random forests
Leo Breiman. 2001 · 2001
Earlier work this paper cites.
An empirical study of the naive Bayes classifier. In IJCAI 2001 workshop on empirical methods in artificial intelligence , Vol. 3. 41–46
Irina Rish et al · 2001
Earlier work this paper cites.
Emotions in everyday life: Probability of occurrence, risk factors, appraisal and reaction patterns
Klaus R Scherer, Tanja Wranik, Janique Sangsue, Véronique Tran, and Ursula Scherer. 2004 · 2004
Earlier work this paper cites.
An introduction to ROC analysis
Tom Fawcett. 2006 · 2006
Earlier work this paper cites.
Sentiment analysis and subjectivity
Bing Liu et al · 2010
Earlier work this paper cites.
Scikit-learn: Machine Learning in Python
F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. 2011 · 2011
Earlier work this paper cites.
A survey of opinion mining and sentiment analysis
Bing Liu and Lei Zhang. 2012 · 2012
Earlier work this paper cites.
Comparative analysis of decision tree classification algorithms
Anuja Priyam, Gupta R Abhijeeta, Anju Rathee, and Saurabh Srivastava. 2013 · 2013
Earlier work this paper cites.
Extracting problematic API features from forum discussions. In 2013 21st International Conference on Program Comprehension (ICPC) . IEEE, 142–151
Yingying Zhang and Daqing Hou. 2013 · 2013
Earlier work this paper cites.
Monitoring sentiment in open source mailing lists: exploratory study on the apache ecosystem.. In CASCON , Vol. 14. 34–44
Parastou Tourani, Yujuan Jiang, and Bram Adams. 2014 · 2014
Earlier work this paper cites.
SimApp: A framework for detecting similar mobile applications by online kernel learning. In Proceedings of the eighth ACM international conference on web search and data mining . 305–314
Ning Chen, Steven CH Hoi, Shaohua Li, and Xiaokui Xiao. 2015 · 2015
Earlier work this paper cites.
Choosing your weapons: On sentiment analysis tools for software engineering research. In 2015 IEEE international conference on software maintenance and evolution (ICSME) . IEEE, 531–535
Robbert Jongeling, Subhajit Datta, and Alexander Serebrenik. 2015 · 2015
Earlier work this paper cites.
The emotional side of software developers in JIRA. In Proceedings of the 13th international conference on mining software repositories . 480–483
Marco Ortu, Alessandro Murgia, Giuseppe Destefanis, Parastou Tourani, Roberto Tonelli, Michele Marchesi, and Bram Adams. 2016 · 2016
Earlier work this paper cites.
Release planning of mobile apps based on user reviews. In Proceedings of the 38th International Conference on Software Engineering . 14–24
Lorenzo Villarroel, Gabriele Bavota, Barbara Russo, Rocco Oliveto, and Massimiliano Di Penta. 2016 · 2016
Earlier work this paper cites.
SentiCR: A customized sentiment analysis tool for code review interactions. In 2017 32nd IEEE/ACM International Conference on Automated Software Engineering (ASE) . IEEE, 106–111
Toufique Ahmed, Amiangshu Bosu, Anindya Iqbal, and Shahram Rahimi. 2017 · 2017
Earlier work this paper cites.
On negative results when using sentiment analysis tools for software engineering research
Robbert Jongeling, Proshanta Sarkar, Subhajit Datta, and Alexander Serebrenik. 2017 · 2017
Earlier work this paper cites.
A survey on hate speech detection using natural language processing. In Proceedings of the fifth international workshop on natural language processing for social media . 1–10
Anna Schmidt and Michael Wiegand. 2017 · 2017
Earlier work this paper cites.
Attention is all you need
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017 · 2017
Earlier work this paper cites.
Sentiment polarity detection for software development. In Proceedings of the 40th International Conference on Software Engineering . 128–128
Fabio Calefato, Filippo Lanubile, Federico Maiorano, and Nicole Novielli. 2018 · 2018
Earlier work this paper cites.
SentiStrength-SE: Exploiting domain specificity for improved sentiment analysis in software engineering text
Md Rakibul Islam and Minhaz F Zibran. 2018b · 2018
Earlier work this paper cites.
Sentiment analysis for software engineering: How far can we go?. In Proceedings of the 40th international conference on software engineering . 94–104
Bin Lin, Fiorella Zampetti, Gabriele Bavota, Massimiliano Di Penta, Michele Lanza, and Rocco Oliveto. 2018 · 2018
Earlier work this paper cites.
An exploratory qualitative and quantitative analysis of emotions in issue report comments of open source systems
Alessandro Murgia, Marco Ortu, Parastou Tourani, Bram Adams, and Serge Demeyer. 2018 · 2018
Earlier work this paper cites.
A benchmark study on sentiment analysis for software engineering research. In Proceedings of the 15th International Conference on Mining Software Repositories . 364–375
Nicole Novielli, Daniela Girardi, and Filippo Lanubile. 2018 · 2018
Earlier work this paper cites.

Similar

Emtk-the emotion mining toolkit. In 2019 IEEE/ACM 4th International Workshop on Emotion Awareness in Software Engineering (SEmotion) . IEEE, 34–37
Fabio Calefato, Filippo Lanubile, Nicole Novielli, and Luigi Quaranta. 2019 · 2019
Cited alongside, same era.
Sentimoji: an emoji-powered learning approach for sentiment analysis in software engineering. In Proceedings of the 2019 27th ACM joint meeting on european software engineering conference and symposium on the foundations of software engineering . 841–852
Zhenpeng Chen, Yanbin Cao, Xuan Lu, Qiaozhu Mei, and Xuanzhe Liu. 2019 · 2019
Cited alongside, same era.
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In Proceedings of NAACL-HLT . 4171–4186
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019 · 2019
Cited alongside, same era.
Sentiment analysis in social media and its application: Systematic literature review
Zulfadzli Drus and Haliyana Khalid. 2019 · 2019
Cited alongside, same era.
MarValous: Machine learning based detection of emotions in the valence-arousal space in software engineering text. In Proceedings of the 34th ACM/SIGAPP Symposium on Applied Computing . 1786–1793
Md Rakibul Islam, Md Kauser Ahmmed, and Minhaz F Zibran. 2019 · 2019
Cited alongside, same era.
Roberta: A robustly optimized bert pretraining approach
Original
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019 · 2019
Cited alongside, same era.
DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter
Original
Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. 2019 · 2019
Cited alongside, same era.
Xlnet: Generalized autoregressive pretraining for language understanding
Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Russ R Salakhutdinov, and Quoc V Le. 2019 · 2019
Cited alongside, same era.
Achieving reliable sentiment analysis in the software engineering domain using bert. In 2020 IEEE International conference on software maintenance and evolution (ICSME) . IEEE, 162–173
Eeshita Biswas, Mehmet Efruz Karabulut, Lori Pollock, and K Vijay-Shanker. 2020 · 2020
Cited alongside, same era.
ALBERT: A Lite BERT for Self-supervised Learning of Language Representations
Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. 2020 · 2020
Cited alongside, same era.
Sentiment analysis: Mining opinions, sentiments, and emotions
Bing Liu. 2020 · 2020
Cited alongside, same era.
Can we use se-specific sentiment analysis tools in a cross-platform setting?. In Proceedings of the 17th International Conference on Mining Software Repositories . 158–168
Nicole Novielli, Fabio Calefato, Davide Dongiovanni, Daniela Girardi, and Filippo Lanubile. 2020 · 2020
Cited alongside, same era.

Then

Data Augmentation for Improving Emotion Recognition in Software Engineering Communication. In Proceedings of the 37th IEEE/ACM International Conference on Automated Software Engineering . 1–13
Mia Mohammad Imran, Yashasvi Jain, Preetha Chatterjee, and Kostadin Damevski. 2022 · 2022
Later among the works it cites.
Opinion mining for software development: a systematic literature review
Bin Lin, Nathan Cassee, Alexander Serebrenik, Gabriele Bavota, Nicole Novielli, and Michele Lanza. 2022 · 2022
Later among the works it cites.
Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivity. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) . 8086–8098
Yao Lu, Max Bartolo, Alastair Moore, Sebastian Riedel, and Pontus Stenetorp. 2022 · 2022
Later among the works it cites.
Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing . 11048–11064
Sewon Min, Xinxi Lyu, Ari Holtzman, Mikel Artetxe, Mike Lewis, Hannaneh Hajishirzi, and Luke Zettlemoyer. 2022 · 2022
Later among the works it cites.
Sentiment analysis tools in software engineering: A systematic mapping study
Martin Obaidi, Lukas Nagel, Alexander Specht, and Jil Klünder. 2022 · 2022
Later among the works it cites.
Opt: Open pre-trained transformer language models
Original
Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al · 2022
Later among the works it cites.
A survey on aspect-based sentiment analysis: Tasks, methods, and challenges
Wenxuan Zhang, Xin Li, Yang Deng, Lidong Bing, and Wai Lam. 2022b · 2022
Later among the works it cites.
Vicuna: An Open-Source Chatbot Impressing GPT-4 with 90%* ChatGPT Quality
Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. 2023 · 2023
Closest in time.
LLMs to the Moon? Reddit Market Sentiment Analysis with Large Language Models. In Companion Proceedings of the ACM Web Conference 2023 . 1014–1019
Xiang Deng, Vasilisa Bashlovkina, Feng Han, Simon Baumgartner, and Michael Bendersky. 2023 · 2023
Closest in time.
Multi-modal api recommendation. In 2023 IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER) . IEEE, 272–283
Ivana Clairine Irsan, Ting Zhang, Ferdian Thung, Kisub Kim, and David Lo. 2023 · 2023
Closest in time.
State of what art? a call for multi-prompt llm evaluation
Original
Moran Mizrahi, Guy Kaplan, Dan Malkin, Rotem Dror, Dafna Shahaf, and Gabriel Stanovsky. 2023 · 2023
Closest in time.
A Sequence-to-Structure Approach to Document-level Targeted Sentiment Analysis. In Findings of the Association for Computational Linguistics: EMNLP 2023 . 7687–7698
Nan Song, Hongjie Cai, Rui Xia, Jianfei Yu, Zhen Wu, and Xinyu Dai. 2023 · 2023
Closest in time.
Stanford Alpaca: An Instruction-following LLaMA model
Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. 2023 · 2023
Closest in time.
Llama: Open and efficient foundation language models
Original
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al · 2023
Closest in time.
Llama 2: Open foundation and fine-tuned chat models
Original
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al · 2023
Closest in time.
Wizardlm: Empowering large language models to follow complex instructions
Original
Can Xu, Qingfeng Sun, Kai Zheng, Xiubo Geng, Pu Zhao, Jiazhan Feng, Chongyang Tao, and Daxin Jiang. 2023 · 2023
Closest in time.
Sentiment Analysis in the Era of Large Language Models: A Reality Check
Original
Wenxuan Zhang, Yue Deng, Bing Liu, Sinno Jialin Pan, and Lidong Bing. 2023 · 2023
Closest in time.
Automatic semantic augmentation of language model prompts (for code summarization). In Proceedings of the IEEE/ACM 46th International Conference on Software Engineering . 1–13
Toufique Ahmed, Kunal Suresh Pai, Premkumar Devanbu, and Earl Barr. 2024 · 2024
Closest in time.
Prompting Is All You Need: Automated Android Bug Replay with Large Language Models. In Proceedings of the 46th IEEE/ACM International Conference on Software Engineering . 1–13
Sidong Feng and Chunyang Chen. 2024 · 2024
Closest in time.
Large language models are few-shot summarizers: Multi-intent comment generation via in-context learning. In Proceedings of the 46th IEEE/ACM International Conference on Software Engineering . 1–13
Mingyang Geng, Shangwen Wang, Dezun Dong, Haotian Wang, Ge Li, Zhi Jin, Xiaoguang Mao, and Xiangke Liao. 2024 · 2024
Closest in time.
Llmparser: An exploratory study on using large language models for log parsing. In Proceedings of the IEEE/ACM 46th International Conference on Software Engineering . 1–13
Zeyang Ma, An Ran Chen, Dong Jae Kim, Tse-Hsun Chen, and Shaowei Wang. 2024 · 2024
Closest in time.
GPT4Rec: Graph Prompt Tuning for Streaming Recommendation. In Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval . 1774–1784
Peiyan Zhang, Yuchen Yan, Xi Zhang, Liying Kang, Chaozhuo Li, Feiran Huang, Senzhang Wang, and Sunghun Kim. 2024 · 2024
Closest in time.
Large Language Model for Vulnerability Detection: Emerging Results and Future Directions. In Proceedings of the 2024 ACM/IEEE 44th International Conference on Software Engineering: New Ideas and Emerging Results (Lisbon, Portugal) (ICSE-NIER’24) . Association for Computing Machinery, New York, NY, USA, 47–51
Xin Zhou, Ting Zhang, and David Lo. 2024 · 2024
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…