Injecting Numerical Reasoning Skills into Language Models

Understand

Large pre-trained language models (LMs) are known to encode substantial amounts of linguistic information.

• However, high-level reasoning skills, such as numerical reasoning, are difficult to learn from a language-modeling objective only.
• Consequently, existing models for numerical reasoning have used specialized architectures with limited flexibility.
• In this work, we show that numerical reasoning is amenable to automatic data generation, and thus one can inject this skill into pre-trained LMs, by generating large amounts of data, and training in a multi-task setup.