Towards Inadequately Pre-trained Models in Transfer Learning

Understand

Pre-training has been a popular learning paradigm in deep learning era, especially in annotation-insufficient scenario.

• Better ImageNet pre-trained models have been demonstrated, from the perspective of architecture, by previous research to have better transferability to downstream tasks.
• However, in this paper, we found that during the same pre-training process, models at middle epochs, which is inadequately pre-trained, can outperform fully trained models when used as feature extractors (FE), while the fine-tuning (FT) performance still grows with the source performance.
• This reveals that there is not a solid positive correlation between top-1 accuracy on ImageNet and the transferring result on target data.