Semantics-aware BERT for Language Understanding

Understand

The latest work on language representations carefully integrates contextualized features into language model training, which enables a series of success especially in various machine reading comprehension and natural language inference tasks.

• However, the existing language representation models including ELMo, GPT and BERT only exploit plain context-sensitive features such as character or word embeddings.
• They rarely consider incorporating structured semantic information which can provide rich semantics for language representation.
• To promote natural language understanding, we propose to incorporate explicit contextual semantics from pre-trained semantic role labeling, and introduce an improved language representation model, Semantics-aware BERT (SemBERT), which is capable of explicitly absorbing contextual semantics over a BERT backbone.