Fetching the paper…

Reading the bibliography…

2023

The Reasonable Effectiveness of Diverse Evaluation Data

Aroyo, Lora, Diaz, Mark, Homan, Christopher et al.

Original
PDF
Post on X

Understand

In this paper, we present findings from an semi-experimental exploration of rater diversity and its influence on safety annotations of conversations generated by humans talking to a generative AI-chat bot.

We find significant differences in judgments produced by raters from different geographic regions and annotation platforms, and correlate these perspectives with demographic sub-groups.
Our work helps define best practices in model development -- specifically human evaluation of generative models -- on the backdrop of growing work on sociotechnical AI evaluations.

The Reasonable Effectiveness of Diverse Evaluation Data · Around

Built on

The unreasonable effectiveness of data
Alon Halevy, Peter Norvig, and Fernando Pereira · 2009
Earlier work this paper cites.
Truth is a lie: Crowd truth and the seven myths of human annotation
Lora Aroyo and Chris Welty · 2015
Earlier work this paper cites.
Addressing age-related bias in sentiment analysis
Mark Díaz, Isaac Johnson, Amanda Lazar, Anne Marie Piper, and Darren Gergle · 2018
Earlier work this paper cites.

Similar

Detecting stance in media on global warming
Original
Yiwei Luo, Dallas Card, and Dan Jurafsky · 2020
Cited alongside, same era.
On releasing annotator-level labels and information in datasets
Vinodkumar Prabhakaran, Aida Mostafazadeh Davani, and Mark Diaz · 2021
Cited alongside, same era.

Then

"everyone wants to do the model work, not the data work": Data cascades in high-stakes ai
Nithya Sambasivan, Shivani Kapania, Hannah Highfill, Diana Akrong, Praveen Kumar Paritosh, and Lora Aroyo · 2021
Later among the works it cites.
Is your toxicity my toxicity? exploring the impact of rater identity on toxicity annotation
Original
Nitesh Goyal, Ian Kivlichan, Rachel Rosen, and Lucy Vasserman · 2022
Later among the works it cites.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…