Evaluating Commonsense in Pre-trained Language Models

Understand

Contextualized representations trained over large raw text data have given remarkable improvements for NLP tasks including question answering and reading comprehension.

• There have been works showing that syntactic, semantic and word sense knowledge are contained in such representations, which explains why they benefit such tasks.
• However, relatively little work has been done investigating commonsense knowledge contained in contextualized representations, which is crucial for human question answering and reading comprehension.
• We study the commonsense ability of GPT, BERT, XLNet, and RoBERTa by testing them on seven challenging benchmarks, finding that language modeling and its variants are effective objectives for promoting models' commonsense ability while bi-directional context and larger training set are bonuses.