TurboTransformers: An Efficient GPU Serving System For Transformer Models

Understand

The transformer is the most critical algorithm innovation of the Nature Language Processing (NLP) field in recent years.

• Unlike the Recurrent Neural Network (RNN) models, Transformers can process on dimensions of sequence lengths in parallel, therefore leading to better accuracy on long sequences.
• However, efficient deployments of them for online services in data centers equipped with GPUs are not easy.
• First, more computation introduced by transformer structures makes it more challenging to meet the latency and throughput constraints of serving.