Magic Pyramid: Accelerating Inference with Early Exiting and Token Pruning

Understand

Pre-training and then fine-tuning large language models is commonly used to achieve state-of-the-art performance in natural language processing (NLP) tasks.

• However, most pre-trained models suffer from low inference speed.
• Deploying such large models to applications with latency constraints is challenging.
• In this work, we focus on accelerating the inference via conditional computations.