Spell Once, Summon Anywhere: A Two-Level Open-Vocabulary Language Model

Understand

We show how the spellings of known words can help us deal with unknown words in open-vocabulary NLP tasks.

• The method we propose can be used to extend any closed-vocabulary generative model, but in this paper we specifically consider the case of neural language modeling.
• Our Bayesian generative story combines a standard RNN language model (generating the word tokens in each sentence) with an RNN-based spelling model (generating the letters in each word type).
• These two RNNs respectively capture sentence structure and word structure, and are kept separate as in linguistics.