Temporal Effects on Pre-trained Models for Language Processing Tasks

Understand

Keeping the performance of language technologies optimal as time passes is of great practical interest.

• We study temporal effects on model performance on downstream language tasks, establishing a nuanced terminology for such discussion and identifying factors essential to conduct a robust study.
• We present experiments for several tasks in English where the label correctness is not dependent on time and demonstrate the importance of distinguishing between temporal model deterioration and temporal domain adaptation for systems using pre-trained representations.
• We find that depending on the task, temporal model deterioration is not necessarily a concern.