Compositional generalization in semantic parsing with pretrained transformers

Understand

Large-scale pretraining instills large amounts of knowledge in deep neural networks.

• This, in turn, improves the generalization behavior of these models in downstream tasks.
• What exactly are the limits to the generalization benefits of large-scale pretraining? Here, we report observations from some simple experiments aimed at addressing this question in the context of two semantic parsing tasks involving natural language, SCAN and COGS.
• We show that language models pretrained exclusively with non-English corpora, or even with programming language corpora, significantly improve out-of-distribution generalization in these benchmarks, compared with models trained from scratch, even though both benchmarks are English-based.