Time Matters: Examine Temporal Effects on Biomedical Language Models

Understand

Time roots in applying language models for biomedical applications: models are trained on historical data and will be deployed for new or future data, which may vary from training data.

• While increasing biomedical tasks have employed state-of-the-art language models, there are very few studies have examined temporal effects on biomedical models when data usually shifts across development and deployment.
• This study fills the gap by statistically probing relations between language model performance and data shifts across three biomedical tasks.
• We deploy diverse metrics to evaluate model performance, distance methods to measure data drifts, and statistical methods to quantify temporal effects on biomedical language models.