2021

Annotators with Attitudes: How Annotator Beliefs And Identities Bias Toxic Language Detection

Sap, Maarten, Swayamdipta, Swabha, Vianna, Laura et al.

Understand

The perceived toxicity of language can vary based on someone's identity and beliefs, but this variation is often ignored when collecting toxic language datasets, resulting in dataset and model biases.

We seek to understand the who, why, and what behind biases in toxicity annotations.
In two online studies with demographically and politically diverse participants, we investigate the effect of annotator identities (who) and beliefs (why), drawing from social psychology research about hate speech, free speech, racist beliefs, political leaning, and more.
We disentangle what is annotated as toxic by considering posts with three characteristics: anti-Black language, African American English (AAE) dialect, and vulgarity.

Built on

Modern racism, ambivalence, and the modern racism scale
John B McConahay. 1986 · 1986
Earlier work this paper cites.
Averaging correlation coefficients: should fisher’s z transformation be used?
N Clayton Silver and William P Dunlap. 1987 · 1987
Earlier work this paper cites.
The politics of language purism
Björn H Jernudd and Michael J Shapiro. 1989 · 1989
Earlier work this paper cites.
Controlling other people. the impact of power on stereotyping
S T Fiske. 1993 · 1993
Earlier work this paper cites.
Racist speech should be protected by the constitution
D Cole. 1996 · 1996
Earlier work this paper cites.
Language, culture and the assessment of african american children
A G Hilliard. 1997 · 1997
Earlier work this paper cites.
African-american language use: Ideology and so-called obscenity
Arthur K Spears et al. 1998 · 1998
Earlier work this paper cites.
Equity, justice, and altruism
Graham F Wagstaff. 1998 · 1998
Earlier work this paper cites.
Social traditionalism and economic conservatism: Two conservative political ideologies in the united states
Stephen D Johnson and Joseph B Tamney. 2001 · 2001
Earlier work this paper cites.
Hate speech and constitutional protection: Priming values of equality and freedom
Gloria Cowan, Miriam Resendez, Elizabeth Marshall, and Ryan Quist. 2002 · 2002
Earlier work this paper cites.
African American English: A Linguistic Introduction , 8.3.2002 edition edition
Lisa Green. 2002 · 2002
Earlier work this paper cites.
From user-centered to participatory design approaches
Elizabeth Sanders. 2002 · 2002
Earlier work this paper cites.
AAE and variation in teachers’ attitudes: A question of school philosophy?
Renée Blake and Cecilia Cutler. 2003 · 2003
Earlier work this paper cites.
Genetic and environmental influences on human psychological differences
Thomas J. Bouchard Jr. and Matt McGue. 2003 · 2003
Earlier work this paper cites.
Empathy, ways of knowing, and interdependence as mediators of gender differences in attitudes toward hate speech and freedom of speech
Gloria Cowan and Désirée Khatchadourian. 2003 · 2003
Earlier work this paper cites.
The impact of social threat on worldview and ideological attitudes
John Duckitt and Kirstin Fisher. 2003 · 2003
Earlier work this paper cites.
African american vernacular english: phonology
Walter F Edwards. 2004 · 2004
Earlier work this paper cites.
The hierarchical structure of the interpersonal reactivity index
Steven Pulos, Jeff Elison, and Randy Lennon. 2004 · 2004
Earlier work this paper cites.
A new front in the culture war? moral traditionalism and voting behavior in us house elections
Jonathan Knuckey. 2005 · 2005
Earlier work this paper cites.
Racism without racists: Color-blind racism and the persistence of racial inequality in the United States
Eduardo Bonilla-Silva. 2006 · 2006
Earlier work this paper cites.
Value sensitive design and information systems
Batya Friedman, Peter H Kahn, and Alan Borning. 2008 · 2008
Earlier work this paper cites.
Dissing oneself versus dissing rivals: effects of status, personality, and sex on the Short-Term and Long-Term attractiveness of Self-Deprecating and Other-Deprecating humor
Gil Greengross and Geoffrey F Miller. 2008 · 2008
Earlier work this paper cites.
The pragmatics of swearing
Timothy Jay and Kristin Janschewitz. 2008 · 2008
Earlier work this paper cites.
Middle-class african americans: Reactions and attitudes toward african american english
Jacquelyn Rahman. 2008 · 2008
Earlier work this paper cites.
Risk-free racism: Whiteness and so-called free speech
David Gillborn. 2009 · 2009
Earlier work this paper cites.
Rating offensive words in three television program contexts
Barry S. Sapolsky, Daniel M. Shafer, and Barbara K. Kaye. 2010 · 2010
Earlier work this paper cites.
Measuring empathy in the 21st century: Development of an empathy index rooted in social cognitive neuroscience and social justice
Karen E Gerdes, Cynthia A Lietz, and Elizabeth A Segal. 2011 · 2011
Earlier work this paper cites.
Whites see racism as a Zero-Sum game that they are now losing
Michael I Norton and Samuel R Sommers. 2011 · 2011
Earlier work this paper cites.
The impact of social media on children, adolescents, and families
Gwenn Schurgin O’Keeffe, Kathleen Clarke-Pearson, and Council on Communications and Media. 2011 · 2011
Earlier work this paper cites.
Future educators’ perceptions of african american vernacular english (AAVE)
Tempii B Champion, Deirdre Cobb-Roberts, and Linda Bland-Stewart. 2012 · 2012
Earlier work this paper cites.
Predicting the importance of freedom of speech and the perceived harm of hate speech
Daniel M Downs and Gloria Cowan. 2012 · 2012
Earlier work this paper cites.
Swearing methodologically : the (im)politeness of expletives in anonymous commentaries on youtube
Marta Dynel. 2012 · 2012
Earlier work this paper cites.
Modern racism attitudes among white students: The role of dominance and authoritarianism and the mediating effects of racial color-blindness
V Paul Poteat and Lisa B Spanierman. 2012 · 2012
Earlier work this paper cites.
The reappropriation of stigmatizing labels: the reciprocal relationship between power and self-labeling
Adam D Galinsky, Cynthia S Wang, Jennifer A Whitson, Eric M Anicich, Kurt Hugenberg, and Galen V Bodenhausen. 2013 · 2013
Earlier work this paper cites.
The marley hypothesis: Denial of racism reflects ignorance of history
Jessica C Nelson, Glenn Adams, and Phia S Salter. 2013 · 2013
Earlier work this paper cites.
Racism, football fans, and online message boards: How social media has added a new dimension to racist discourse in English football
Jamie Cleland. 2014 · 2014
Earlier work this paper cites.
A solution to the collective action problem in between-group conflict with within-group inequality
Sergey Gavrilets and Laura Fortunato. 2014 · 2014
Earlier work this paper cites.

Similar

The significance of hedonic values for environmentally relevant attitudes, preferences, and actions
Linda Steg, Goda Perlaviciute, Ellen Van der Werff, and Judith Lurvink. 2014 · 2014
Cited alongside, same era.
Speaking up for african american english: Equity and inclusion in early childhood settings
Margaret Beneke and Gregory A Cheatham. 2015 · 2015
Cited alongside, same era.
Group-based differences in perceptions of racism: What counts, to whom, and why?
Evelyn R Carter and Mary C Murphy. 2015 · 2015
Cited alongside, same era.
Sounding black or white: Priming identity and biracial speech
Sarah E Gaither, Ariel M Cohen-Goldberg, Calvin L Gidney, and Keith B Maddox. 2015 · 2015
Cited alongside, same era.
“who are these people?” evaluating the demographic characteristics and political preferences of mturk survey respondents
Connor Huff and Dustin Tingley. 2015 · 2015
Cited alongside, same era.
Demographic dialectal variation in social media: A case study of African-American English
Su Lin Blodgett, Lisa Green, and Brendan O’Connor. 2016 · 2016
Cited alongside, same era.
Analyzing the targets of hate in online social media
Leandro Silva, Mainack Mondal, Denzil Correa, Fabrício Benevenuto, and Ingmar Weber. 2016 · 2016
Cited alongside, same era.
Are you a racist or am I seeing things? annotator influence on hate speech detection on Twitter
Zeerak Waseem. 2016 · 2016
Cited alongside, same era.
Hateful symbols or hateful people? predictive features for hate speech detection on Twitter
Zeerak Waseem and Dirk Hovy. 2016 · 2016
Cited alongside, same era.
The problem with bias: Allocative versus representational harms in machine learning
Solon Barocas, Kate Crawford, Aaron Shapiro, and Hanna Wallach. 2017 · 2017
Cited alongside, same era.
Automated hate speech detection and the problem of offensive language
Thomas Davidson, Dana Warmsley, Michael Macy, and Ingmar Weber. 2017 · 2017
Cited alongside, same era.
Postelection surrealism and nostalgic racism in the hands of donald trump
Donna M Goldstein and Kira Hall. 2017 · 2017
Cited alongside, same era.

Then

Identifying and measuring annotator bias based on annotators’ demographic characteristics
Hala Al Kuwatly, Maximilian Wich, and Georg Groh. 2020 · 2020
Later among the works it cites.
How instagram’s algorithm is censoring women and vulnerable users but helping online abusers
Carolina Are. 2020 · 2020
Later among the works it cites.
Proposals for improved regulation of harmful online content
Susan Benesch. 2020 · 2020
Later among the works it cites.
Language (technology) is power: A critical survey of “bias” in NLP
Su Lin Blodgett, Solon Barocas, Hal Daumé III, and Hanna Wallach. 2020 · 2020
Later among the works it cites.
“this is us”: Free speech embedded in whiteness, racism and coloniality in aotearoa, new zealand
Christine Helen Elers and Pooja Jayan. 2020 · 2020
Later among the works it cites.
Expanding the debate about content moderation: Scholarly research agendas for the coming policy debates
Tarleton Gillespie, Patricia Aufderheide, Elinor Carmi, Ysabel Gerrard, Robert Gorwa, Ariadna Matamoros-Fernandez, Sarah T Roberts, Aram Sinnreich, and Sarah Myers West. 2020 · 2020
Later among the works it cites.
Data science as political action: Grounding data science in a politics of justice
Ben Green. 2020 · 2020
Later among the works it cites.
Fortifying toxic speech detectors against veiled toxicity
Xiaochuang Han and Yulia Tsvetkov. 2020 · 2020
Later among the works it cites.
“sassy queens”: Stylistic orthographic variation in twitter and the enregisterment of AAVE
Christian Ilbury. 2020 · 2020
Later among the works it cites.
Identifying and addressing design and policy challenges in online content moderation
J A Jiang. 2020 · 2020
Later among the works it cites.
Don’t ask if artificial intelligence is good or fair, ask how it shifts power
Pratyusha Kalluri. 2020 · 2020
Later among the works it cites.
Distinction without a difference? an assessment of mturk worker types
Eric Loepp and Jarrod T Kelly. 2020 · 2020
Later among the works it cites.
Reconsidering Reagan: Racism, Republicans, and the Road to Trump
Daniel S Lucks. 2020 · 2020
Later among the works it cites.
The radicalization risks of gpt-3 and advanced neural language models
Original
Kris McGuffie and Alex Newhouse. 2020 · 2020
Later among the works it cites.
How game features give rise to altruism and collective action? implications for cultivating cooperation by gamification
Marc Riar, Benedikt Morschheuser, Juho Hamari, and Rüdiger Zarnekow. 2020 · 2020
Later among the works it cites.
Social bias frames: Reasoning about social and power implications of language
Maarten Sap, Saadia Gabriel, Lianhui Qin, Dan Jurafsky, Noah A. Smith, and Yejin Choi. 2020 · 2020
Later among the works it cites.
Political psycholinguistics: A comprehensive analysis of the language habits of liberal and conservative social media users
Joanna Sterling, John T Jost, and Richard Bonneau. 2020 · 2020
Later among the works it cites.
Language and identity: a study of African American Vernacular English and its status in American society
Ane Ueland. 2020 · 2020
Later among the works it cites.
Whose opinions matter? perspective-aware models to identify opinions of hate speech victims in abusive language detection
Original
Sohail Akhtar, Valerio Basile, and Viviana Patti. 2021 · 2021
Closest in time.
Ground-Truth, whose truth? – examining the challenges with annotating toxic text datasets
Original
Kofi Arhin, Ioana Baldini, Dennis Wei, Karthikeyan Natesan Ramamurthy, and Moninder Singh. 2021 · 2021
Closest in time.
Algorithmic injustice: a relational ethics approach
Abeba Birhane. 2021 · 2021
Closest in time.
Model Positionality: A Novel Framework for Data Science with Subjective Target Concepts
Scott Allen Cambo. 2021 · 2021
Closest in time.
Hate speech classifiers learn human-like social stereotypes
Original
Aida Mostafazadeh Davani, Mohammad Atari, Brendan Kennedy, and Morteza Dehghani. 2021 · 2021
Closest in time.
Beyond black & white: Leveraging annotator disagreement via soft-label multi-task learning
Tommaso Fornaciari, Alexandra Uma, Silviu Paun, Barbara Plank, Dirk Hovy, and Massimo Poesio. 2021 · 2021
Closest in time.
The importance of modeling social factors of language: Theory and practice
Dirk Hovy and Diyi Yang. 2021 · 2021
Closest in time.
Understanding international perceptions of the severity of harmful content online
Jialun Aaron Jiang, Morgan Klaus Scheuerman, Casey Fiesler, and Jed R Brubaker. 2021 · 2021
Closest in time.
Conservatives and incivility
Ashley Muddiman. 2021 · 2021
Closest in time.
On releasing Annotator-Level labels and information in datasets
Original
Vinodkumar Prabhakaran, Aida Mostafazadeh Davani, and Mark Díaz. 2021 · 2021
Closest in time.
The psychological Well-Being of content moderators: The emotional labor of commercial moderation and avenues for improving support
Miriah Steiger, Timir J Bharucha, Sukrit Venkatagiri, Martin J Riedl, and Matthew Lease. 2021 · 2021
Closest in time.
Learning from the worst: Dynamically generated datasets to improve online hate detection
Bertie Vidgen, Tristan Thrush, Zeerak Waseem, and Douwe Kiela. 2021 · 2021
Closest in time.
Disembodied machine learning: On the illusion of objectivity in nlp
Original
Zeerak Waseem, Smarika Lulz, Joachim Bingel, and Isabelle Augenstein. 2021 · 2021
Closest in time.
Sociology of race
Zuleyka Zevallos. 2017 · 2021
Closest in time.
Challenges in automated debiasing for toxic language detection
Xuhui Zhou, Maarten Sap, Swabha Swayamdipta, Yejin Choi, and Noah Smith. 2021 · 2021
Closest in time.
Jury learning: Integrating dissenting voices into machine learning models
Original
Mitchell L Gordon, Michelle S Lam, Joon Sung Park, Kayur Patel, Jeffrey T Hancock, Tatsunori Hashimoto, and Michael S Bernstein. 2022 · 2022
Closest in time.
Social-Cognitive and affective antecedents of code switching and the consequences of linguistic racism for black people and people of color
Darin G Johnson, Bradley D Mattan, Nelson Flores, Nina Lauharatanahirun, and Emily B Falk. 2022 · 2022
Closest in time.
Listening to affected communities to define extreme speech: Dataset and experiments
Antonis Maronikolakis, Axel Wisiorek, Leah Nann, Haris Jabbar, Sahana Udupa, and Hinrich Schütze. 2022 · 2022
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…