Rare Words: A Major Problem for Contextualized Embeddings And How to Fix it by Attentive Mimicking

Understand

Pretraining deep neural network architectures with a language modeling objective has brought large improvements for many natural language processing tasks.

• Exemplified by BERT, a recently proposed such architecture, we demonstrate that despite being trained on huge amounts of data, deep language models still struggle to understand rare words.
• To fix this problem, we adapt Attentive Mimicking, a method that was designed to explicitly learn embeddings for rare words, to deep language models.
• In order to make this possible, we introduce one-token approximation, a procedure that enables us to use Attentive Mimicking even when the underlying language model uses subword-based tokenization, i.e., it does not assign embeddings to all words.