Built on

Measuring massive multitask language understanding
Original
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2020 · 2009
Earlier work this paper cites.
A survey on bias and fairness in machine learning
Ninareh Mehrabi, Fred Morstatter, Nripsuta Saxena, Kristina Lerman, and Aram Galstyan. 2021 · 2021
Earlier work this paper cites.
Alue: Arabic language understanding evaluation
Haitham Seelawi, Ibraheem Tuffaha, Mahmoud Gzawi, Wael Farhan, Bashar Talafha, Riham Badawi, Zyad Sober, Oday Al-Dweik, Abed Alhakim Freihat, and Hussein Al-Natsheh. 2021 · 2021
Earlier work this paper cites.
Synthbio: A case study in human-ai collaborative curation of text datasets
Original
Ann Yuan, Daphne Ippolito, Vitaly Nikolaev, Chris Callison-Burch, Andy Coenen, and Sebastian Gehrmann. 2021 · 2021
Earlier work this paper cites.
Holistic evaluation of language models
Original
Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga, Yian Zhang, Deepak Narayanan, Yuhuai Wu, Ananya Kumar, et al. 2022 · 2022
Earlier work this paper cites.
Chatgpt
OpenAI. 2022 · 2022
Earlier work this paper cites.
Training language models to follow instructions with human feedback, 2022
Original
Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022 · 2022
Earlier work this paper cites.
Cultural incongruencies in artificial intelligence
Original
Vinodkumar Prabhakaran, Rida Qadri, and Ben Hutchinson. 2022 · 2022
Earlier work this paper cites.

Similar

Gpt-4 technical report
Original
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023 · 2023
Cited alongside, same era.
Red-teaming large language models using chain of utterances for safety-alignment
Original
Rishabh Bhardwaj and Soujanya Poria. 2023 · 2023
Cited alongside, same era.
Toxicity in chatgpt: Analyzing persona-assigned language models
Original
Ameet Deshpande, Vishvak Murahari, Tanmay Rajpurohit, Ashwin Kalyan, and Karthik Narasimhan. 2023 · 2023
Cited alongside, same era.
Orca: A challenging benchmark for arabic language understanding
Original
AbdelRahim Elmadany, El Moatez Billah Nagoudi, and Muhammad Abdul-Mageed. 2023 · 2023
Cited alongside, same era.

Then

How trustworthy are open-source llms? an assessment under malicious demonstrations shows their vulnerabilities
Original
Lingbo Mo, Boshi Wang, Muhao Chen, and Huan Sun. 2023 · 2023
Later among the works it cites.
Emojis as anchors to detect arabic offensive language and hate speech
Hamdy Mubarak, Sabit Hassan, and Shammur Absar Chowdhury. 2023 · 2023
Later among the works it cites.
Having beer after prayer? measuring cultural bias in large language models
Original
Tarek Naous, Michael J Ryan, and Wei Xu. 2023 · 2023
Later among the works it cites.
Jais and jais-chat: Arabic-centric foundation and instruction-tuned open generative large language models
Original
Neha Sengupta, Sunil Kumar Sahu, Bokang Jia, Satheesh Katipomu, Haonan Li, Fajri Koto, Osama Mohammed Afzal, Samta Kamboj, Onkar Pandit, Rahul Pal, et al. 2023 · 2023
Later among the works it cites.
Safetybench: Evaluating the safety of large language models with multiple choice questions

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…

2024

AraTrust: An Evaluation of Trustworthiness for LLMs in Arabic

Alghamdi, Emad A., Masoud, Reem I., Alnuhait, Deema et al.

Understand

The swift progress and widespread acceptance of artificial intelligence (AI) systems highlight a pressing requirement to comprehend both the capabilities and potential risks associated with AI.

Given the linguistic complexity, cultural richness, and underrepresented status of Arabic in AI research, there is a pressing need to focus on Large Language Models (LLMs) performance and safety for Arabic-related tasks.
Despite some progress in their development, there is a lack of comprehensive trustworthiness evaluation benchmarks, which presents a major challenge in accurately assessing and improving the safety of LLMs when prompted in Arabic.
In this paper, we introduce AraTrust, the first comprehensive trustworthiness benchmark for LLMs in Arabic.

Built on

Measuring massive multitask language understanding
Original
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2020 · 2009
Earlier work this paper cites.
A survey on bias and fairness in machine learning
Ninareh Mehrabi, Fred Morstatter, Nripsuta Saxena, Kristina Lerman, and Aram Galstyan. 2021 · 2021
Earlier work this paper cites.
Alue: Arabic language understanding evaluation
Haitham Seelawi, Ibraheem Tuffaha, Mahmoud Gzawi, Wael Farhan, Bashar Talafha, Riham Badawi, Zyad Sober, Oday Al-Dweik, Abed Alhakim Freihat, and Hussein Al-Natsheh. 2021 · 2021
Earlier work this paper cites.
Synthbio: A case study in human-ai collaborative curation of text datasets
Original
Ann Yuan, Daphne Ippolito, Vitaly Nikolaev, Chris Callison-Burch, Andy Coenen, and Sebastian Gehrmann. 2021 · 2021
Earlier work this paper cites.
Holistic evaluation of language models
Original
Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga, Yian Zhang, Deepak Narayanan, Yuhuai Wu, Ananya Kumar, et al. 2022 · 2022
Earlier work this paper cites.
Chatgpt
OpenAI. 2022 · 2022
Earlier work this paper cites.
Training language models to follow instructions with human feedback, 2022
Original
Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022 · 2022
Earlier work this paper cites.
Cultural incongruencies in artificial intelligence
Original
Vinodkumar Prabhakaran, Rida Qadri, and Ben Hutchinson. 2022 · 2022
Earlier work this paper cites.

Similar

Gpt-4 technical report
Original
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023 · 2023
Cited alongside, same era.
Red-teaming large language models using chain of utterances for safety-alignment
Original
Rishabh Bhardwaj and Soujanya Poria. 2023 · 2023
Cited alongside, same era.
Toxicity in chatgpt: Analyzing persona-assigned language models
Original
Ameet Deshpande, Vishvak Murahari, Tanmay Rajpurohit, Ashwin Kalyan, and Karthik Narasimhan. 2023 · 2023
Cited alongside, same era.
Orca: A challenging benchmark for arabic language understanding
Original
AbdelRahim Elmadany, El Moatez Billah Nagoudi, and Muhammad Abdul-Mageed. 2023 · 2023
Cited alongside, same era.

Then

How trustworthy are open-source llms? an assessment under malicious demonstrations shows their vulnerabilities
Original
Lingbo Mo, Boshi Wang, Muhao Chen, and Huan Sun. 2023 · 2023
Later among the works it cites.
Emojis as anchors to detect arabic offensive language and hate speech
Hamdy Mubarak, Sabit Hassan, and Shammur Absar Chowdhury. 2023 · 2023
Later among the works it cites.
Having beer after prayer? measuring cultural bias in large language models
Original
Tarek Naous, Michael J Ryan, and Wei Xu. 2023 · 2023
Later among the works it cites.
Jais and jais-chat: Arabic-centric foundation and instruction-tuned open generative large language models
Original
Neha Sengupta, Sunil Kumar Sahu, Bokang Jia, Satheesh Katipomu, Haonan Li, Fajri Koto, Osama Mohammed Afzal, Samta Kamboj, Onkar Pandit, Rahul Pal, et al. 2023 · 2023
Later among the works it cites.
Safetybench: Evaluating the safety of large language models with multiple choice questions

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Acegpt, localizing large language models in arabic

Original

Huang Huang, Fei Yu, Jianqing Zhu, Xuening Sun, Hao Cheng, Dingjie Song, Zhihong Chen, Abdulmohsen Alharthi, Bang An, Ziche Liu, et al. 2023 · 2023

Cited alongside, same era.

GPTAraEval: A comprehensive evaluation of ChatGPT on Arabic NLP

Md Tawkat Islam Khondaker, Abdul Waheed, El Moatez Billah Nagoudi, and Muhammad Abdul-Mageed. 2023 · 2023

Cited alongside, same era.

Multi-step jailbreaking privacy attacks on chatgpt

Original

Haoran Li, Dadi Guo, Wei Fan, Mingshi Xu, and Yangqiu Song. 2023 · 2023

Cited alongside, same era.

Trustworthy llms: a survey and guideline for evaluating large language models’ alignment

Original

Yang Liu, Yuanshun Yao, Jean-Francois Ton, Xiaoying Zhang, Ruocheng Guo Hao Cheng, Yegor Klochkov, Muhammad Faaiz Taufiq, and Hang Li. 2023 · 2023

Cited alongside, same era.

Decodingtrust: A comprehensive assessment of trustworthiness in gpt models

Original

Boxin Wang, Weixin Chen, Hengzhi Pei, Chulin Xie, Mintong Kang, Chenhui Zhang, Chejian Xu, Zidi Xiong, Ritik Dutta, Rylan Schaeffer, et al. 2023a

Cited in the paper.

All languages matter: On the multilingual safety of large language models

Original

Wenxuan Wang, Zhaopeng Tu, Chang Chen, Youliang Yuan, Jen-tse Huang, Wenxiang Jiao, and Michael R Lyu. 2023b

Cited in the paper.

Do-not-answer: A dataset for evaluating safeguards in llms

Original

Yuxia Wang, Haonan Li, Xudong Han, Preslav Nakov, and Timothy Baldwin. 2023c

Cited in the paper.

Cvalues: Measuring the values of chinese large language models from safety to responsibility

Original

Guohai Xu, Jiayi Liu, Ming Yan, Haotian Xu, Jinghui Si, Zhuoran Zhou, Peng Yi, Xing Gao, Jitao Sang, Rong Zhang, et al. 2023a

Cited in the paper.

Original

Zhexin Zhang, Leqi Lei, Lindong Wu, Rui Sun, Yongkang Huang, Chong Long, Xiao Liu, Xuanyu Lei, Jie Tang, and Minlie Huang. 2023 · 2023

Later among the works it cites.

Promptbench: Towards evaluating the robustness of large language models on adversarial prompts

Original

Kaijie Zhu, Jindong Wang, Jiaheng Zhou, Zichen Wang, Hao Chen, Yidong Wang, Linyi Yang, Wei Ye, Neil Zhenqiang Gong, Yue Zhang, et al. 2023 · 2023

Later among the works it cites.

Trustllm: Trustworthiness in large language models

Original

Lichao Sun, Yue Huang, Haoran Wang, Siyuan Wu, Qihui Zhang, Chujie Gao, Yixin Huang, Wenhan Lyu, Yixuan Zhang, Xiner Li, et al. 2024 · 2024

Closest in time.