SemEval-2023 Task 11: Learning With Disagreements (LeWiDi)

Understand

NLP datasets annotated with human judgments are rife with disagreements between the judges.

• This is especially true for tasks depending on subjective judgments such as sentiment analysis or offensive language detection.
• Particularly in these latter cases, the NLP community has come to realize that the approach of 'reconciling' these different subjective interpretations is inappropriate.
• Many NLP researchers have therefore concluded that rather than eliminating disagreements from annotated corpora, we should preserve them-indeed, some argue that corpora should aim to preserve all annotator judgments.