Exposing ambiguities in a relation-extraction gold standard with crowdsourcing

Understand

Semantic relation extraction is one of the frontiers of biomedical natural language processing research.

• Gold standards are key tools for advancing this research.
• It is challenging to generate these standards because of the high cost of expert time and the difficulty in establishing agreement between annotators.
• We implemented and evaluated a microtask crowdsourcing approach that can produce a gold standard for extracting drug-disease relations.