ERNIE 3.0 Titan: Exploring Larger-scale Knowledge Enhanced Pre-training for Language Understanding and Generation

Understand

Pre-trained language models have achieved state-of-the-art results in various Natural Language Processing (NLP) tasks.

• GPT-3 has shown that scaling up pre-trained language models can further exploit their enormous potential.
• A unified framework named ERNIE 3.0 was recently proposed for pre-training large-scale knowledge enhanced models and trained a model with 10 billion parameters.
• ERNIE 3.0 outperformed the state-of-the-art models on various NLP tasks.