Measuring the Reliability of Hate Speech Annotations: The Case of the European Refugee Crisis

Understand

Some users of social media are spreading racist, sexist, and otherwise hateful content.

• For the purpose of training a hate speech detection system, the reliability of the annotations is crucial, but there is no universally agreed-upon definition.
• We collected potentially hateful messages and asked two groups of internet users to determine whether they were hate speech or not, whether they should be banned or not and to rate their degree of offensiveness.
• One of the groups was shown a definition prior to completing the survey.