What does BERT Learn from Multiple-Choice Reading Comprehension Datasets?

Understand

Multiple-Choice Reading Comprehension (MCRC) requires the model to read the passage and question, and select the correct answer among the given options.

• Recent state-of-the-art models have achieved impressive performance on multiple MCRC datasets.
• However, such performance may not reflect the model's true ability of language understanding and reasoning.
• In this work, we adopt two approaches to investigate what BERT learns from MCRC datasets: 1) an un-readable data attack, in which we add keywords to confuse BERT, leading to a significant performance drop; and 2) an un-answerable data training, in which we train BERT on partial or shuffled input.