The Effect of Natural Distribution Shift on Question Answering Models

Understand

We build four new test sets for the Stanford Question Answering Dataset (SQuAD) and evaluate the ability of question-answering systems to generalize to new data.

• Our first test set is from the original Wikipedia domain and measures the extent to which existing systems overfit the original test set.
• Despite several years of heavy test set re-use, we find no evidence of adaptive overfitting.
• The remaining three test sets are constructed from New York Times articles, Reddit posts, and Amazon product reviews and measure robustness to natural distribution shifts.