Character-Aware Neural Language Models

Understand

We describe a simple neural language model that relies only on character-level inputs.

• Predictions are still made at the word-level.
• Our model employs a convolutional neural network (CNN) and a highway network over characters, whose output is given to a long short-term memory (LSTM) recurrent neural network language model (RNN-LM).
• On the English Penn Treebank the model is on par with the existing state-of-the-art despite having 60% fewer parameters.