Why Do Masked Neural Language Models Still Need Common Sense Knowledge?

Understand

Currently, contextualized word representations are learned by intricate neural network models, such as masked neural language models (MNLMs).

• The new representations significantly enhanced the performance in automated question answering by reading paragraphs.
• However, identifying the detailed knowledge trained in the MNLMs is difficult owing to numerous and intermingled parameters.
• This paper provides empirical but insightful analyses on the pretrained MNLMs with respect to common sense knowledge.