Numeracy for Language Models: Evaluating and Improving their Ability to Predict Numbers

Understand

Numeracy is the ability to understand and work with numbers.

• It is a necessary skill for composing and understanding documents in clinical, scientific, and other technical domains.
• In this paper, we explore different strategies for modelling numerals with language models, such as memorisation and digit-by-digit composition, and propose a novel neural architecture that uses a continuous probability density function to model numerals from an open vocabulary.
• Our evaluation on clinical and scientific datasets shows that using hierarchical models to distinguish numerals from words improves a perplexity metric on the subset of numerals by 2 and 4 orders of magnitude, respectively, over non-hierarchical models.