Assessing the Benchmarking Capacity of Machine Reading Comprehension Datasets

Understand

Existing analysis work in machine reading comprehension (MRC) is largely concerned with evaluating the capabilities of systems.

• However, the capabilities of datasets are not assessed for benchmarking language understanding precisely.
• We propose a semi-automated, ablation-based methodology for this challenge; By checking whether questions can be solved even after removing features associated with a skill requisite for language understanding, we evaluate to what degree the questions do not require the skill.
• Experiments on 10 datasets (e.g., CoQA, SQuAD v2.0, and RACE) with a strong baseline model show that, for example, the relative scores of a baseline model provided with content words only and with shuffled sentence words in the context are on average 89.2% and 78.5% of the original score, respectively.