What to do about non-standard (or non-canonical) language in NLP

Understand

Real world data differs radically from the benchmark corpora we use in natural language processing (NLP).

• As soon as we apply our technologies to the real world, performance drops.
• The reason for this problem is obvious: NLP models are trained on samples from a limited set of canonical varieties that are considered standard, most prominently English newswire.
• However, there are many dimensions, e.g., socio-demographics, language, genre, sentence type, etc.