Are you tough enough? Framework for Robustness Validation of Machine Comprehension Systems

Understand

Deep Learning NLP domain lacks procedures for the analysis of model robustness.

• In this paper we propose a framework which validates robustness of any Question Answering model through model explainers.
• We propose that a robust model should transgress the initial notion of semantic similarity induced by word embeddings to learn a more human-like understanding of meaning.
• We test this property by manipulating questions in two ways: swapping important question word for 1) its semantically correct synonym and 2) for word vector that is close in embedding space.