Are Pre-trained Convolutions Better than Pre-trained Transformers?

Understand

In the era of pre-trained language models, Transformers are the de facto choice of model architectures.

• While recent research has shown promise in entirely convolutional, or CNN, architectures, they have not been explored using the pre-train-fine-tune paradigm.
• In the context of language models, are convolutional models competitive to Transformers when pre-trained? This paper investigates this research question and presents several interesting findings.
• Across an extensive set of experiments on 8 datasets/tasks, we find that CNN-based pre-trained models are competitive and outperform their Transformer counterpart in certain scenarios, albeit with caveats.