2023

Taqyim: Evaluating Arabic NLP Tasks Using ChatGPT Models

Alyafeai, Zaid, Alshaibani, Maged S., AlKhamissi, Badr et al.

Understand

Large language models (LLMs) have demonstrated impressive performance on various downstream tasks without requiring fine-tuning, including ChatGPT, a chat-based model built on top of LLMs such as GPT-3.5 and GPT-4.

Despite having a lower training proportion compared to English, these models also exhibit remarkable capabilities in other languages.
In this study, we assess the performance of GPT-3.5 and GPT-4 models on seven distinct Arabic NLP tasks: sentiment analysis, translation, transliteration, paraphrasing, part of speech tagging, summarization, and diacritization.
Our findings reveal that GPT-4 outperforms GPT-3.5 on five out of the seven tasks.

Built on

Language models are few-shot learners
Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · 1901
Earlier work this paper cites.
75 languages, 1 model: Parsing universal dependencies universally
Original
Kondratyuk, D. and Straka, M. (2019) · 1904
Earlier work this paper cites.
Roberta: A robustly optimized bert pretraining approach
Original
Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., and Stoyanov, V. (2019) · 1907
Earlier work this paper cites.
Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension
Original
Lewis, M., Liu, Y., Goyal, N., Ghazvininejad, M., Mohamed, A., Levy, O., Stoyanov, V., and Zettlemoyer, L. (2019) · 1910
Earlier work this paper cites.
Prague arabic dependency treebank: A word on the million words
Smrz, O., Bielickỳ, V., Kourilová, I., Krácmar, J., Hajic, J., and Zemánek, P. (2008) · 2008
Earlier work this paper cites.
Arabic diacritic restoration approach based on maximum entropy models
Zitouni, I. and Sarikaya, R. (2009) · 2009
Earlier work this paper cites.
Using mechanical turk to create a corpus of arabic summaries
El-Haj, M., Kruschwitz, U., and Fox, C. (2010) · 2010
Earlier work this paper cites.
Wikilingua: A new benchmark dataset for cross-lingual abstractive summarization
Original
Ladhak, F., Durmus, E., Cardie, C., and McKeown, K. (2020) · 2010
Earlier work this paper cites.
Transliteration of Arabizi into Arabic orthography: Developing a parallel annotated Arabizi-Arabic script SMS/chat corpus
Bies, A., Song, Z., Maamouri, M., Grimes, S., Lee, H., Wright, J., Strassel, S., Habash, N., Eskander, R., and Rambow, O. (2014) · 2014
Earlier work this paper cites.
The united nations parallel corpus v1. 0
Ziemski, M., Junczys-Dowmunt, M., and Pouliquen, B. (2016) · 2016
Earlier work this paper cites.
Arabic tweets sentimental analysis using machine learning
Alomari, K. M., ElSherif, H. M., and Shaalan, K. (2017) · 2017
Earlier work this paper cites.
Arabic diacritization: Stats, rules, and hacks
Darwish, K., Mubarak, H., and Abdelali, A. (2017) · 2017
Earlier work this paper cites.
Deep contextualized word representations
Peters, M. E., Neumann, M., Iyyer, M., Gardner, M., Clark, C., Lee, K., and Zettlemoyer, L. (2018) · 2018
Earlier work this paper cites.
Bert: Pre-training of deep bidirectional transformers for language understanding
Original
Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K. (2019) · 2019
Earlier work this paper cites.
Highly effective Arabic diacritization using sequence to sequence modeling
Mubarak, H., Abdelali, A., Sajjad, H., Samih, Y., and Darwish, K. (2019) · 2019
Earlier work this paper cites.
Language models are unsupervised multitask learners
Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., Sutskever, I., et al. (2019) · 2019
Earlier work this paper cites.

Similar

Arbert & marbert: deep bidirectional transformers for arabic
Original
Abdul-Mageed, M., Elmadany, A., and Nagoudi, E. M. B. (2020) · 2020
Cited alongside, same era.
Deep diacritization: Efficient hierarchical recurrence for improved Arabic diacritization
AlKhamissi, B., ElNokrashy, M., and Gabr, M. (2020) · 2020
Cited alongside, same era.
A unified model for arabizi detection and transliteration using sequence-to-sequence models
Shazal, A., Usman, A., and Habash, N. (2020) · 2020
Cited alongside, same era.
Universal dependencies 2.7
Zeman et al., D. (2020) · 2020
Cited alongside, same era.
Building arabic paraphrasing benchmark based on transformation rules
Alian, M., Awajan, A., Al-Hasan, A., and Akuzhia, R. (2021) · 2021
Cited alongside, same era.
On the opportunities and risks of foundation models
Original
Bommasani et al., R. (2021) · 2021
Cited alongside, same era.
Multitask prompted training enables zero-shot task generalization
Original
Sanh, V., Webson, A., Raffel, C., Bach, S. H., Sutawika, L., Alyafeai, Z., Chaffin, A., Stiegler, A., Scao, T. L., Raja, A., et al. (2021) · 2021
Cited alongside, same era.
Palm: Scaling language modeling with pathways
Original
Chowdhery, A., Narang, S., Devlin, J., Bosma, M., Mishra, G., Roberts, A., Barham, P., Chung, H. W., Sutton, C., Gehrmann, S., et al. (2022) · 2022
Cited alongside, same era.
Revisiting pre-trained language models and their evaluation for arabic natural language processing
Ghaddar, A., Wu, Y., Bagga, S., Rashid, A., Bibi, K., Rezagholizadeh, M., Xing, C., Wang, Y., Duan, X., Wang, Z., et al. (2022) · 2022
Cited alongside, same era.
How well does chatgpt do when taking the medical licensing exams? the implications of large language models for medical education and knowledge assessment
Gilson, A., Safranek, C., Huang, T., Socrates, V., Chi, L., Taylor, R. A., and Chartash, D. (2022) · 2022
Cited alongside, same era.
Training compute-optimal large language models
Original
Hoffmann, J., Borgeaud, S., Mensch, A., Buchatskaya, E., Cai, T., Rutherford, E., Casas, D. d. L., Hendricks, L. A., Welbl, J., Clark, A., et al. (2022) · 2022
Cited alongside, same era.
Trip: Triangular document-level pre-training for multilingual language models
Original
Lu, H., Huang, H., Ma, S., Zhang, D., Lam, W., and Wei, F. (2022) · 2022
Cited alongside, same era.

Then

Aramus: Pushing the limits of data and model scale for arabic natural language processing
Original
Alghamdi, A., Duan, X., Jiang, W., Wang, Z., Wu, Y., Xia, Q., Wang, Z., Zheng, Y., Rezagholizadeh, M., Huai, B., et al. (2023) · 2023
Closest in time.
A multitask, multilingual, multimodal evaluation of chatgpt on reasoning, hallucination, and interactivity
Original
Bang, Y., Cahyawijaya, S., Lee, N., Dai, W., Su, D., Wilie, B., Lovenia, H., Ji, Z., Yu, T., Chung, W., et al. (2023) · 2023
Closest in time.
Evaluating the feasibility of chatgpt in healthcare: An analysis of multiple clinical and research scenarios
Cascella, M., Montomoli, J., Bellini, V., and Elena, B. (2023) · 2023
Closest in time.
Chatgpt evaluation on sentence level relations: A focus on temporal, causal, and discourse relations
Original
Chan, C., Cheng, J., Wang, W., Jiang, Y., Fang, T., Liu, X., and Song, Y. (2023) · 2023
Closest in time.
Mathematics, word problems, common sense, and artificial intelligence
Original
Davis, E. (2023) · 2023
Closest in time.
Mathematical capabilities of chatgpt
Original
Frieder, S., Pinchetti, L., Griffiths, R.-R., Salvatori, T., Lukasiewicz, T., Petersen, P. C., Chevalier, A., and Berner, J. (2023) · 2023
Closest in time.
How close is chatgpt to human experts? comparison corpus, evaluation, and detection
Original
Guo, B., Zhang, X., Wang, Z., Jiang, M., Nie, J., Ding, Y., Yue, J., and Wu, Y. (2023) · 2023
Closest in time.
How good are gpt models at machine translation? a comprehensive evaluation
Hendy, A., Abdelrehim, M., Sharaf, A., Raunak, V., Gabr, M., Matsushita, H., Kim, Y. J., Afify, M., and Awadalla, H. H. (2023) · 2023
Closest in time.
Is chatgpt a good translator? yes with gpt-4 as the engine
Original
Jiao, W., Wang, W., Huang, J., Wang, X., and Tu, Z. (2023) · 2023
Closest in time.
Summary of chatgpt/gpt-4 research and perspective towards the future of large language models
Liu, Y., Han, T., Ma, S., Zhang, J., Yang, Y., Tian, J., He, H., Li, A., He, M., Liu, Z., Wu, Z., Zhu, D., Li, X., Qiang, N., Shen, D., Liu, T., and Ge, B. (2023) · 2023
Closest in time.
Is chatgpt a general-purpose natural language processing task solver?
Original
Qin, C., Zhang, A., Zhang, Z., Chen, J., Yasunaga, M., and Yang, D. (2023) · 2023
Closest in time.
Gptaraeval: A comprehensive evaluation of chatgpt on arabic nlp
Tawkat Islam Khondaker, M., Waheed, A., Moatez Billah Nagoudi, E., and Abdul-Mageed, M. (2023) · 2023
Closest in time.
Llama: Open and efficient foundation language models
Original
Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., et al. (2023) · 2023
Closest in time.
A survey of large language models
Original
Zhao, W. X., Zhou, K., Li, J., Tang, T., Wang, X., Hou, Y., Min, Y., Zhang, B., Zhang, J., Dong, Z., et al. (2023) · 2023
Closest in time.
Are chatgpt and other pretrained language models good parasitologists?
Šlapeta, J. (2023) · 2023
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…