Which Model Shall I Choose? Cost/Quality Trade-offs for Text Classification Tasks

Understand

Industry practitioners always face the problem of choosing the appropriate model for deployment under different considerations, such as to maximize a metric that is crucial for production, or to reduce the total cost given financial concerns.

• In this work, we focus on the text classification task and present a quantitative analysis for this challenge.
• Using classification accuracy as the main metric, we evaluate the classifiers' performances for a variety of models, including large language models, along with their associated costs, including the annotation cost, training (fine-tuning) cost, and inference cost.
• We then discuss the model choices for situations like having a large number of samples needed for inference.