Performance Control in Early Exiting to Deploy Large Models at the Same Cost of Smaller Ones

Understand

Early Exiting (EE) is a promising technique for speeding up inference by adaptively allocating compute resources to data points based on their difficulty.

• The approach enables predictions to exit at earlier layers for simpler samples while reserving more computation for challenging ones.
• In this study, we first present a novel perspective on the EE approach, showing that larger models deployed with EE can achieve higher performance than smaller models while maintaining similar computational costs.
• As existing EE approaches rely on confidence estimation at each exit point, we further study the impact of overconfidence on the controllability of the compute-performance trade-off.