BabyBear: Cheap inference triage for expensive language models

Understand

Transformer language models provide superior accuracy over previous models but they are computationally and environmentally expensive.

• Borrowing the concept of model cascading from computer vision, we introduce BabyBear, a framework for cascading models for natural language processing (NLP) tasks to minimize cost.
• The core strategy is inference triage, exiting early when the least expensive model in the cascade achieves a sufficiently high-confidence prediction.
• We test BabyBear on several open source data sets related to document classification and entity recognition.