LIFT: Language-Interfaced Fine-Tuning for Non-Language Machine Learning Tasks

Understand

Fine-tuning pretrained language models (LMs) without making any architectural changes has become a norm for learning various language downstream tasks.

• However, for non-language downstream tasks, a common practice is to employ task-specific designs for input, output layers, and loss functions.
• For instance, it is possible to fine-tune an LM into an MNIST classifier by replacing the word embedding layer with an image patch embedding layer, the word token output layer with a 10-way output layer, and the word prediction loss with a 10-way classification loss, respectively.
• A natural question arises: Can LM fine-tuning solve non-language downstream tasks without changing the model architecture or loss function? To answer this, we propose Language-Interfaced Fine-Tuning (LIFT) and study its efficacy and limitations by conducting an extensive empirical study on a suite of non-language classification and regression tasks.