2023

Is ChatGPT a Good Sentiment Analyzer? A Preliminary Study

Wang, Zengzhi, Xie, Qiming, Feng, Yi et al.

Understand

Recently, ChatGPT has drawn great attention from both the research community and the public.

We are particularly interested in whether it can serve as a universal sentiment analyzer.
To this end, in this work, we provide a preliminary evaluation of ChatGPT on the understanding of \emph{opinions}, \emph{sentiments}, and \emph{emotions} contained in the text.
Specifically, we evaluate it in three settings, including \emph{standard} evaluation, \emph{polarity shift} evaluation and \emph{open-domain} evaluation.

Built on

Mining and summarizing customer reviews
Minqing Hu and Bing Liu. 2004 · 2004
Earlier work this paper cites.
Opinion observer: analyzing and comparing opinions on the web
Bing Liu, Minqing Hu, and Junsheng Cheng. 2005 · 2005
Earlier work this paper cites.
Mining comparative sentences and relations
Nitin Jindal and Bing Liu. 2006 · 2006
Earlier work this paper cites.
Sentiment classification and polarity shifting
Shoushan Li, Sophia Y. M. Lee, Ying Chen, Chu-Ren Huang, and Guodong Zhou. 2010 · 2010
Earlier work this paper cites.
Sentence and expression level annotation of opinions in user-generated discourse
Cigdem Toprak, Niklas Jakob, and Iryna Gurevych. 2010 · 2010
Earlier work this paper cites.
Semantic multidimensional scaling for open-domain sentiment analysis
Erik Cambria, Yangqiu Song, Haixun Wang, and Newton Howard. 2012 · 2012
Earlier work this paper cites.
Sentiment analysis and opinion mining
Bing Liu. 2012 · 2012
Earlier work this paper cites.
Recursive deep models for semantic compositionality over a sentiment treebank
Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Ng, and Christopher Potts. 2013 · 2013
Earlier work this paper cites.
Adaptive recursive neural network for target-dependent Twitter sentiment classification
Li Dong, Furu Wei, Chuanqi Tan, Duyu Tang, Ming Zhou, and Ke Xu. 2014 · 2014
Earlier work this paper cites.
A corpus of comparisons in product reviews
Wiltrud Kessler and Jonas Kuhn. 2014 · 2014
Earlier work this paper cites.
SemEval-2014 task 4: Aspect based sentiment analysis
Maria Pontiki, Dimitris Galanis, John Pavlopoulos, Harris Papageorgiou, Ion Androutsopoulos, and Suresh Manandhar. 2014 · 2014
Earlier work this paper cites.
Sentiment Analysis - Mining Opinions, Sentiments, and Emotions
Bing Liu. 2015 · 2015
Earlier work this paper cites.
Neural networks for open domain targeted sentiment
Meishan Zhang, Yue Zhang, and Duy-Tin Vo. 2015 · 2015
Earlier work this paper cites.
Event-driven emotion cause extraction with corpus construction
Lin Gui, Dongyin Wu, Ruifeng Xu, Qin Lu, and Yu Zhou. 2016 · 2016
Earlier work this paper cites.
Polarity shift detection, elimination and ensemble: A three-stage model for document-level sentiment analysis
Rui Xia, Feng Xu, Jianfei Yu, Yong Qi, and Erik Cambria. 2016 · 2016
Earlier work this paper cites.
Deep reinforcement learning from human preferences
Paul F. Christiano, Jan Leike, Tom B. Brown, Miljan Martic, Shane Legg, and Dario Amodei. 2017 · 2017
Earlier work this paper cites.
GLUE: A multi-task benchmark and analysis platform for natural language understanding
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2018 · 2018
Earlier work this paper cites.
Deep learning for sentiment analysis: A survey
Lei Zhang, Shuai Wang, and Bing Liu. 2018 · 2018
Earlier work this paper cites.
BERT: Pre-training of deep bidirectional transformers for language understanding
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019 · 2019
Earlier work this paper cites.
From independent prediction to reordered prediction: Integrating relative position and global label information to emotion cause identification
Zixiang Ding, Huihui He, Mengran Zhang, and Rui Xia. 2019 · 2019
Earlier work this paper cites.
Exploiting BERT for end-to-end aspect-based sentiment analysis
Xin Li, Lidong Bing, Wenxuan Zhang, and Wai Lam. 2019 · 2019
Earlier work this paper cites.

Similar

Emotion-cause pair extraction: A new task to emotion analysis in texts
Rui Xia and Zixiang Ding. 2019 · 2019
Cited alongside, same era.
Language models are few-shot learners
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020 · 2020
Cited alongside, same era.
ECPE-2D: emotion-cause pair extraction based on joint two-dimensional representation, interaction and prediction
Zixiang Ding, Rui Xia, and Jianfei Yu. 2020 · 2020
Cited alongside, same era.
Exploring the limits of transfer learning with a unified text-to-text transformer
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020 · 2020
Cited alongside, same era.
Sentiment analysis using deep learning architectures: a review
Ashima Yadav and Dinesh Kumar Vishwakarma. 2020 · 2020
Cited alongside, same era.
Learning implicit sentiment in aspect-based sentiment analysis with supervised contrastive pre-training
Zhengyan Li, Yicheng Zou, Chong Zhang, Qi Zhang, and Zhongyu Wei. 2021 · 2021
Cited alongside, same era.
Comparative opinion quintuple extraction from product reviews
Ziheng Liu, Rui Xia, and Jianfei Yu. 2021 · 2021
Cited alongside, same era.
Multi-task learning of negation and speculation for targeted sentiment classification
Andrew Moore and Jeremy Barnes. 2021 · 2021
Cited alongside, same era.
Scaling language models: Methods, analysis & insights from training gopher
Original
Jack W. Rae, Sebastian Borgeaud, Trevor Cai, Katie Millican, Jordan Hoffmann, H. Francis Song, John Aslanides, Sarah Henderson, Roman Ring, Susannah Young, Eliza Rutherford, Tom Hennigan, Jacob Menick, Albin Cassirer, Richard Powell, George van den Driessche, Lisa Anne Hendricks, Maribeth Rauh, Po-Sen Huang, Amelia Glaese, Johannes Welbl, Sumanth Dathathri, Saffron Huang, Jonathan Uesato, John Mellor, Irina Higgins, Antonia Creswell, Nat McAleese, Amy Wu, Erich Elsen, Siddhant M. Jayakumar, Elena Buchatskaya, David Budden, Esme Sutherland, Karen Simonyan, Michela Paganini, Laurent Sifre, Lena Martens, Xiang Lorraine Li, Adhiguna Kuncoro, Aida Nematzadeh, Elena Gribovskaya, Domenic Donato, Angeliki Lazaridou, Arthur Mensch, Jean-Baptiste Lespiau, Maria Tsimpoukelli, Nikolai Grigorev, Doug Fritz, Thibault Sottiaux, Mantas Pajarskas, Toby Pohlen, Zhitao Gong, Daniel Toyama, Cyprien de Masson d’Autume, Yujia Li, Tayfun Terzi, Vladimir Mikulik, Igor Babuschkin, Aidan Clark, Diego de Las Casas, Aurelia Guy, Chris Jones, James Bradbury, Matthew J. Johnson, Blake A. Hechtman, Laura Weidinger, Iason Gabriel, William S. Isaac, Edward Lockhart, Simon Osindero, Laura Rimell, Chris Dyer, Oriol Vinyals, Kareem Ayoub, Jeff Stanway, Lorrayne Bennett, Demis Hassabis, Koray Kavukcuoglu, and Geoffrey Irving. 2021 · 2021
Cited alongside, same era.
Towards generative aspect-based sentiment analysis
Wenxuan Zhang, Xin Li, Yang Deng, Lidong Bing, and Wai Lam. 2021 · 2021
Cited alongside, same era.
Palm: Scaling language modeling with pathways
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam Shazeer, Vinodkumar Prabhakaran, Emily Reif, Nan Du, Ben Hutchinson, Reiner Pope, James Bradbury, Jacob Austin, Michael Isard, Guy Gur-Ari, Pengcheng Yin, Toju Duke, Anselm Levskaya, Sanjay Ghemawat, Sunipa Dev, Henryk Michalewski, Xavier Garcia, Vedant Misra, Kevin Robinson, Liam Fedus, Denny Zhou, Daphne Ippolito, David Luan, Hyeontaek Lim, Barret Zoph, Alexander Spiridonov, Ryan Sepassi, David Dohan, Shivani Agrawal, Mark Omernick, Andrew M. Dai, Thanumalayan Sankaranarayana Pillai, Marie Pellat, Aitor Lewkowycz, Erica Moreira, Rewon Child, Oleksandr Polozov, Katherine Lee, Zongwei Zhou, Xuezhi Wang, Brennan Saeta, Mark Diaz, Orhan Firat, Michele Catasta, Jason Wei, Kathy Meier-Hellstern, Douglas Eck, Jeff Dean, Slav Petrov, and Noah Fiedel. 2022 · 2022
Cited alongside, same era.
Inheriting the wisdom of predecessors: A multiplex cascade framework for unified aspect-based sentiment analysis
Hao Fei, Fei Li, Chenliang Li, Shengqiong Wu, Jingye Li, and Donghong Ji. 2022 · 2022
Cited alongside, same era.

Then

Lamda: Language models for dialog applications
Original
Romal Thoppilan, Daniel De Freitas, Jamie Hall, Noam Shazeer, Apoorv Kulshreshtha, Heng-Tze Cheng, Alicia Jin, Taylor Bos, Leslie Baker, Yu Du, YaGuang Li, Hongrae Lee, Huaixiu Steven Zheng, Amin Ghafouri, Marcelo Menegali, Yanping Huang, Maxim Krikun, Dmitry Lepikhin, James Qin, Dehao Chen, Yuanzhong Xu, Zhifeng Chen, Adam Roberts, Maarten Bosma, Yanqi Zhou, Chung-Ching Chang, Igor Krivokon, Will Rusch, Marc Pickett, Kathleen S. Meier-Hellstern, Meredith Ringel Morris, Tulsee Doshi, Renelito Delos Santos, Toju Duke, Johnny Soraker, Ben Zevenbergen, Vinodkumar Prabhakaran, Mark Diaz, Ben Hutchinson, Kristen Olson, Alejandra Molina, Erin Hoffman-John, Josh Lee, Lora Aroyo, Ravi Rajakumar, Alena Butryna, Matthew Lamm, Viktoriya Kuzmina, Joe Fenton, Aaron Cohen, Rachel Bernstein, Ray Kurzweil, Blaise Aguera-Arcas, Claire Cui, Marian Croak, Ed H. Chi, and Quoc Le. 2022 · 2022
Later among the works it cites.
Self-consistency improves chain of thought reasoning in language models
Original
Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc V. Le, Ed H. Chi, and Denny Zhou. 2022 · 2022
Later among the works it cites.
Chain of thought prompting elicits reasoning in large language models
Original
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed H. Chi, Quoc Le, and Denny Zhou. 2022 · 2022
Later among the works it cites.
The unreliability of explanations in few-shot prompting for textual reasoning
Xi Ye and Greg Durrett. 2022 · 2022
Later among the works it cites.
Towards unifying the label space for aspect- and sentence-based sentiment analysis
Yiming Zhang, Min Zhang, Sai Wu, and Junbo Zhao. 2022c · 2022
Later among the works it cites.
Mets-cov: A dataset of medical entity and targeted sentiment on COVID-19 related tweets
Original
Peilin Zhou, Zeqiang Wang, Dading Chong, Zhijiang Guo, Yining Hua, Zichang Su, Zhiyang Teng, Jiageng Wu, and Jie Yang. 2022 · 2022
Later among the works it cites.
A multitask, multilingual, multimodal evaluation of chatgpt on reasoning, hallucination, and interactivity
Original
Yejin Bang, Samuel Cahyawijaya, Nayeon Lee, Wenliang Dai, Dan Su, Bryan Wilie, Holy Lovenia, Ziwei Ji, Tiezheng Yu, Willy Chung, Quyet V. Do, Yan Xu, and Pascale Fung. 2023 · 2023
Closest in time.
A categorical archive of chatgpt failures
Original
Ali Borji. 2023 · 2023
Closest in time.
Mathematical capabilities of chatgpt
Original
Simon Frieder, Luca Pinchetti, Ryan-Rhys Griffiths, Tommaso Salvatori, Thomas Lukasiewicz, Philipp Christian Petersen, Alexis Chevalier, and Julius Berner. 2023 · 2023
Closest in time.
How close is chatgpt to human experts? comparison corpus, evaluation, and detection
Original
Biyang Guo, Xin Zhang, Ziyuan Wang, Minqi Jiang, Jinran Nie, Yuxuan Ding, Jianwei Yue, and Yupeng Wu. 2023 · 2023
Closest in time.
Is chatgpt A good translator? A preliminary study
Original
Wenxiang Jiao, Wenxuan Wang, Jen-tse Huang, Xing Wang, and Zhaopeng Tu. 2023 · 2023
Closest in time.
A systematic study and comprehensive evaluation of chatgpt on benchmark datasets
Original
Md. Tahmid Rahman Laskar, M. Saiful Bari, Mizanur Rahman, Md Amran Hossen Bhuiyan, Shafiq Joty, and Jimmy Xiangji Huang. 2023 · 2023
Closest in time.
GPT-4 technical report
Original
OpenAI. 2023 · 2023
Closest in time.
REPLUG: retrieval-augmented black-box language models
Original
Weijia Shi, Sewon Min, Michihiro Yasunaga, Minjoon Seo, Rich James, Mike Lewis, Luke Zettlemoyer, and Wen-tau Yih. 2023 · 2023
Closest in time.
Llama: Open and efficient foundation language models
Original
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurélien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023 · 2023
Closest in time.
On the robustness of chatgpt: An adversarial and out-of-distribution perspective
Original
Jindong Wang, Xixu Hu, Wenxin Hou, Hao Chen, Runkai Zheng, Yidong Wang, Linyi Yang, Haojun Huang, Wei Ye, Xiubo Geng, Binxing Jiao, Yue Zhang, and Xing Xie. 2023 · 2023
Closest in time.
A comprehensive capability analysis of GPT-3 and GPT-3.5 series models
Original
Junjie Ye, Xuanting Chen, Nuo Xu, Can Zu, Zekai Shao, Shichun Liu, Yuhan Cui, Zeyang Zhou, Chao Gong, Yang Shen, Jie Zhou, Siming Chen, Tao Gui, Qi Zhang, and Xuanjing Huang. 2023 · 2023
Closest in time.
Can chatgpt understand too? A comparative study on chatgpt and fine-tuned BERT
Original
Qihuang Zhong, Liang Ding, Juhua Liu, Bo Du, and Dacheng Tao. 2023 · 2023
Closest in time.
Exploring AI ethics of chatgpt: A diagnostic analysis
Original
Terry Yue Zhuo, Yujin Huang, Chunyang Chen, and Zhenchang Xing. 2023 · 2023
Closest in time.
Text Data Mining
Chengqing Zong, Rui Xia, and Jiajun Zhang. 2021 · 2023
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…