A linearized framework and a new benchmark for model selection for fine-tuning

Understand

Fine-tuning from a collection of models pre-trained on different domains (a "model zoo") is emerging as a technique to improve test accuracy in the low-data regime.

• However, model selection, i.e.
• how to pre-select the right model to fine-tune from a model zoo without performing any training, remains an open topic.
• We use a linearized framework to approximate fine-tuning, and introduce two new baselines for model selection -- Label-Gradient and Label-Feature Correlation.