The future is different: Large pre-trained language models fail in prediction tasks

Understand

Large pre-trained language models (LPLM) have shown spectacular success when fine-tuned on downstream supervised tasks.

• Yet, it is known that their performance can drastically drop when there is a distribution shift between the data used during training and that used at inference time.
• In this paper we focus on data distributions that naturally change over time and introduce four new REDDIT datasets, namely the WALLSTREETBETS, ASKSCIENCE, THE DONALD, and POLITICS sub-reddits.
• First, we empirically demonstrate that LPLM can display average performance drops of about 88% (in the best case!) when predicting the popularity of future posts from sub-reddits whose topic distribution changes with time.