Learning Semantic Representations for Novel Words: Leveraging Both Form and Context

Understand

Word embeddings are a key component of high-performing natural language processing (NLP) systems, but it remains a challenge to learn good representations for novel words on the fly, i.e., for words that did not occur in the training data.

• The general problem setting is that word embeddings are induced on an unlabeled training corpus and then a model is trained that embeds novel words into this induced embedding space.
• Currently, two approaches for learning embeddings of novel words exist: (i) learning an embedding from the novel word's surface-form (e.g., subword n-grams) and (ii) learning an embedding from the context in which it occurs.
• In this paper, we propose an architecture that leverages both sources of information - surface-form and context - and show that it results in large increases in embedding quality.