DQ-BART: Efficient Sequence-to-Sequence Model via Joint Distillation and Quantization

Understand

Large-scale pre-trained sequence-to-sequence models like BART and T5 achieve state-of-the-art performance on many generative NLP tasks.

• However, such models pose a great challenge in resource-constrained scenarios owing to their large memory requirements and high latency.
• To alleviate this issue, we propose to jointly distill and quantize the model, where knowledge is transferred from the full-precision teacher model to the quantized and distilled low-precision student model.
• Empirical analyses show that, despite the challenging nature of generative tasks, we were able to achieve a 16.5x model footprint compression ratio with little performance drop relative to the full-precision counterparts on multiple summarization and QA datasets.