A Framework for Evaluation of Machine Reading Comprehension Gold Standards

Understand

Machine Reading Comprehension (MRC) is the task of answering a question over a paragraph of text.

• While neural MRC systems gain popularity and achieve noticeable performance, issues are being raised with the methodology used to establish their performance, particularly concerning the data design of gold standards that are used to evaluate them.
• There is but a limited understanding of the challenges present in this data, which makes it hard to draw comparisons and formulate reliable hypotheses.
• As a first step towards alleviating the problem, this paper proposes a unifying framework to systematically investigate the present linguistic features, required reasoning and background knowledge and factual correctness on one hand, and the presence of lexical cues as a lower bound for the requirement of understanding on the other hand.