Mengzi: Towards Lightweight yet Ingenious Pre-trained Models for Chinese

Understand

Although pre-trained models (PLMs) have achieved remarkable improvements in a wide range of NLP tasks, they are expensive in terms of time and resources.

• This calls for the study of training more efficient models with less computation but still ensures impressive performance.
• Instead of pursuing a larger scale, we are committed to developing lightweight yet more powerful models trained with equal or less computation and friendly to rapid deployment.
• This technical report releases our pre-trained model called Mengzi, which stands for a family of discriminative, generative, domain-specific, and multimodal pre-trained model variants, capable of a wide range of language and vision tasks.