2023

GPTAraEval: A Comprehensive Evaluation of ChatGPT on Arabic NLP

Khondaker, Md Tawkat Islam, Waheed, Abdul, Nagoudi, El Moatez Billah et al.

Understand

ChatGPT's emergence heralds a transformative phase in NLP, particularly demonstrated through its excellent performance on many English benchmarks.

However, the model's efficacy across diverse linguistic contexts remains largely uncharted territory.
This work aims to bridge this knowledge gap, with a primary focus on assessing ChatGPT's capabilities on Arabic languages and dialectal varieties.
Our comprehensive study conducts a large-scale automated and human evaluation of ChatGPT, encompassing 44 distinct language understanding and generation tasks on over 60 different datasets.

Built on

Anetac: Arabic named entity transliteration and classification dataset
Original
Mohamed Seghir Hadj Ameur, Farid Meziane, and Ahmed Guessoum. 2019 · 1907
Earlier work this paper cites.
RoBERTa: A Robustly Optimized BERT Pretraining Approach
Original
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019 · 1907
Earlier work this paper cites.
Better evaluation for grammatical error correction
Daniel Dahlmeier and Hwee Tou Ng. 2012 · 2012
Earlier work this paper cites.
A multidialectal parallel corpus of arabic
Houda Bouamor, Nizar Habash, and Kemal Oflazer. 2014 · 2014
Earlier work this paper cites.
The first QALB shared task on automatic text correction for Arabic
Behrang Mohit, Alla Rozovskaya, Nizar Habash, Wajdi Zaghouani, and Ossama Obeid. 2014 · 2014
Earlier work this paper cites.
Arabic Dialect Identification
Omar F Zaidan and Chris Callison-Burch. 2014 · 2014
Earlier work this paper cites.
The united nations parallel corpus v1. 0
Michal Ziemski, Marcin Junczys-Dowmunt, and Bruno Pouliquen. 2016 · 2016
Earlier work this paper cites.
Dailydialog: A manually labelled multi-turn dialogue dataset
Yanran Li, Hui Su, Xiaoyu Shen, Wenjie Li, Ziqiang Cao, and Shuzi Niu. 2017 · 2017
Earlier work this paper cites.
Integrating stance detection and fact checking in a unified corpus
Ramy Baly, Mitra Mohtarami, James Glass, Lluís Màrquez, Alessandro Moschitti, and Preslav Nakov. 2018 · 2018
Earlier work this paper cites.
Xnli: Evaluating cross-lingual sentence representations
Original
Alexis Conneau, Ruty Rinott, Guillaume Lample, Adina Williams, Samuel R. Bowman, Holger Schwenk, and Veselin Stoyanov. 2018 · 2018
Earlier work this paper cites.
SemEval-2018 task 1: Affect in tweets
Saif Mohammad, Felipe Bravo-Marquez, Mohammad Salameh, and Svetlana Kiritchenko. 2018 · 2018
Earlier work this paper cites.
The MADAR shared task on Arabic fine-grained dialect identification
Houda Bouamor, Sabit Hassan, and Nizar Habash. 2019 · 2019
Earlier work this paper cites.
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019 · 2019
Earlier work this paper cites.
How contextual are contextualized word representations? Comparing the geometry of BERT, ELMo, and GPT-2 embeddings
Kawin Ethayarajh. 2019 · 2019
Earlier work this paper cites.
Arabic text diacritization using deep neural networks
Original
Ali Fadel, Ibraheem Tuffaha, Bara’ Al-Jawarneh, and Mahmoud Al-Ayyoub. 2019 · 2019
Earlier work this paper cites.
IDAT@FIRE2019: Overview of the Track on Irony Detection in Arabic Tweets.
Bilal Ghanem, Jihen Karoui, Farah Benamara, Véronique Moriceau, and Paolo Rosso. 2019 · 2019
Earlier work this paper cites.
Nsurl-2019 task 8: Semantic question similarity in arabic
Haitham Seelawi, Ahmad Mustafa, Hesham Al-Bataineh, Wael Farhan, and Hussein T Al-Natsheh. 2019 · 2019
Earlier work this paper cites.
Glue: A multi-task benchmark and analysis platform for natural language understanding
Original
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. 2019 · 2019
Earlier work this paper cites.
Arabic Dialect Identification in the Wild
Ahmed Abdelali, Hamdy Mubarak, Younes Samih, Sabit Hassan, and Kareem Darwish. 2020 · 2020
Earlier work this paper cites.
NADI 2020: The first nuanced Arabic dialect identification shared task
Muhammad Abdul-Mageed, Chiyu Zhang, Houda Bouamor, and Nizar Habash. 2020a · 2020
Earlier work this paper cites.
Understanding and detecting dangerous speech in social media
Ali Alshehri, El Moatez Billah Nagoudi, and Muhammad Abdul-Mageed. 2020 · 2020
Earlier work this paper cites.
On the Cross-lingual Transferability of Monolingual Representations
Mikel Artetxe, Sebastian Ruder, and Dani Yogatama. 2020 · 2020
Earlier work this paper cites.
Language models are few-shot learners
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020 · 2020
Earlier work this paper cites.

Similar

Habibi-a multi dialect multi national arabic song lyrics corpus
Mahmoud El-Haj. 2020 · 2020
Cited alongside, same era.
From Arabic Sentiment Analysis to Sarcasm Detection: The ArSarcasm Dataset
Ibrahim Abu Farha and Walid Magdy. 2020 · 2020
Cited alongside, same era.
Stance prediction and claim verification: An Arabic perspective
Jude Khouja. 2020 · 2020
Cited alongside, same era.
On the sentence embeddings from pre-trained language models
Bohan Li, Hao Zhou, Junxian He, Mingxuan Wang, Yiming Yang, and Lei Li. 2020 · 2020
Cited alongside, same era.
Overview of OSACT4 Arabic offensive language detection shared task
Hamdy Mubarak, Kareem Darwish, Walid Magdy, Tamer Elsayed, and Hend Al-Khalifa. 2020 · 2020
Cited alongside, same era.
Machine generation and detection of Arabic manipulated and fake news
El Moatez Billah Nagoudi, AbdelRahim Elmadany, Muhammad Abdul-Mageed, and Tariq Alhindi. 2020 · 2020
Cited alongside, same era.
Adversarial nli: A new benchmark for natural language understanding
Original
Yixin Nie, Adina Williams, Emily Dinan, Mohit Bansal, Jason Weston, and Douwe Kiela. 2020 · 2020
Cited alongside, same era.
TaPaCo: A corpus of sentential paraphrases for 73 languages
Yves Scherrer. 2020 · 2020
Cited alongside, same era.
Benchmarking transformer-based language models for Arabic sentiment and sarcasm detection
Ibrahim Abu Farha and Walid Magdy. 2021 · 2021
Cited alongside, same era.
Arabic gloss wsd using bert
Mohammed El-Razzaz, Mohamed Waleed Fakhr, and Fahima A Maghraby. 2021 · 2021
Cited alongside, same era.
Xl-sum: Large-scale multilingual abstractive summarization for 44 languages
Original
Tahmid Hasan, Abhik Bhattacharjee, Md Saiful Islam, Kazi Samin, Yuan-Fang Li, Yong-Bin Kang, M. Sohel Rahman, and Rifat Shahriyar. 2021 · 2021
Cited alongside, same era.
Adult content detection on Arabic Twitter: Analysis and experiments
Hamdy Mubarak, Sabit Hassan, and Ahmed Abdelali. 2021 · 2021
Cited alongside, same era.

Then

Opt: Open pre-trained transformer language models
Original
Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. 2022 · 2022
Later among the works it cites.
A multitask, multilingual, multimodal evaluation of chatgpt on reasoning, hallucination, and interactivity
Original
Yejin Bang, Samuel Cahyawijaya, Nayeon Lee, Wenliang Dai, Dan Su, Bryan Wilie, Holy Lovenia, Ziwei Ji, Tiezheng Yu, Willy Chung, Quyet V. Do, Yan Xu, and Pascale Fung. 2023 · 2023
Closest in time.
Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality
Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. 2023 · 2023
Closest in time.
Orca: A challenging benchmark for arabic language understanding
AbdelRahim Elmadany, El Moatez Billah Nagoudi, and Muhammad Abdul-Mageed. 2023 · 2023
Closest in time.
How to design translation prompts for chatgpt: An empirical study
Original
Yuan Gao, Ruili Wang, and Feng Hou. 2023 · 2023
Closest in time.
Chatgpt outperforms crowd-workers for text-annotation tasks
Original
Fabrizio Gilardi, Meysam Alizadeh, and Maël Kubli. 2023 · 2023
Closest in time.
How good are gpt models at machine translation? a comprehensive evaluation
Original
Amr Hendy, Mohamed Abdelrehim, Amr Sharaf, Vikas Raunak, Mohamed Gabr, Hitokazu Matsushita, Young Jin Kim, Mohamed Afify, and Hany Hassan Awadalla. 2023 · 2023
Closest in time.
Not all languages are created equal in llms: Improving multilingual capability by cross-lingual-thought prompting
Original
Haoyang Huang, Tianyi Tang, Dongdong Zhang, Wayne Xin Zhao, Ting Song, Yan Xia, and Furu Wei. 2023 · 2023
Closest in time.
Is chatgpt a good translator? yes with gpt-4 as the engine
Original
Wenxiang Jiao, Wenxuan Wang, Jen tse Huang, Xing Wang, and Zhaopeng Tu. 2023 · 2023
Closest in time.
Evaluating gpt-4 and chatgpt on japanese medical licensing examinations
Original
Jungo Kasai, Yuhei Kasai, Keisuke Sakaguchi, Yutaro Yamada, and Dragomir Radev. 2023 · 2023
Closest in time.
Chatgpt beyond english: Towards a comprehensive evaluation of large language models in multilingual learning
Original
Viet Dac Lai, Nghia Trung Ngo, Amir Pouran Ben Veyseh, Hieu Man, Franck Dernoncourt, Trung Bui, and Thien Huu Nguyen. 2023 · 2023
Closest in time.
A systematic study and comprehensive evaluation of chatgpt on benchmark datasets
Original
Md Tahmid Rahman Laskar, M Saiful Bari, Mizanur Rahman, Md Amran Hossen Bhuiyan, Shafiq R. Joty, and J. Huang. 2023 · 2023
Closest in time.
Open-domain response generation in low-resource settings using self-supervised pre-training of warm-started transformers
Tarek Naous, Zahraa Bassyouni, Bassel Mousi, Hazem Hajj, Wassim El Hajj, and Khaled Shaban. 2023 · 2023
Closest in time.
Chatgpt versus traditional question answering for knowledge graphs: Current status and future directions towards knowledge graph chatbots
Original
Reham Omar, Omij Mangukiya, Panos Kalnis, and Essam Mansour. 2023 · 2023
Closest in time.
Towards making the most of chatgpt for machine translation
Original
Keqin Peng, Liang Ding, Qihuang Zhong, Li Shen, Xuebo Liu, Min Zhang, Yuanxin Ouyang, and Dacheng Tao. 2023 · 2023
Closest in time.
Testing the reliability of chatgpt for text annotation and classification: A cautionary remark
Original
Michael V. Reiss. 2023 · 2023
Closest in time.
In chatgpt we trust? measuring and characterizing the reliability of chatgpt
Original
Xinyue Shen, Zeyuan Chen, Michael Backes, and Yang Zhang. 2023 · 2023
Closest in time.
Evaluation of chatgpt as a question answering system for answering complex questions
Original
Yiming Tan, Dehai Min, Yu Li, Wenbo Li, Nan Hu, Yongrui Chen, and Guilin Qi. 2023 · 2023
Closest in time.
A prompt pattern catalog to enhance prompt engineering with chatgpt
Original
Jules White, Quchen Fu, Sam Hays, Michael Sandborn, Carlos Olea, Henry Gilbert, Ashraf Elnashar, Jesse Spencer-Smith, and Douglas C. Schmidt. 2023 · 2023
Closest in time.
Why does chatgpt fall short in answering questions faithfully?
Original
Shen Zheng, Jie Huang, and Kevin Chen-Chuan Chang. 2023 · 2023
Closest in time.
Lima: Less is more for alignment
Original
Chunting Zhou, Pengfei Liu, Puxin Xu, Srini Iyer, Jiao Sun, Yuning Mao, Xuezhe Ma, Avia Efrat, Ping Yu, Lili Yu, Susan Zhang, Gargi Ghosh, Mike Lewis, Luke Zettlemoyer, and Omer Levy. 2023 · 2023
Closest in time.
Multilingual machine translation with large language models: Empirical results and analysis
Original
Wenhao Zhu, Hongyi Liu, Qingxiu Dong, Jingjing Xu, Shujian Huang, Lingpeng Kong, Jiajun Chen, and Lei Li. 2023 · 2023
Closest in time.
Can large language models transform computational social science?
Original
Caleb Ziems, William Held, Omar Shaikh, Jiaao Chen, Zhehao Zhang, and Diyi Yang. 2023 · 2023
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…