Is neural language acquisition similar to natural? A chronological probing study

Understand

The probing methodology allows one to obtain a partial representation of linguistic phenomena stored in the inner layers of the neural network, using external classifiers and statistical analysis.

• Pre-trained transformer-based language models are widely used both for natural language understanding (NLU) and natural language generation (NLG) tasks making them most commonly used for downstream applications.
• However, little analysis was carried out, whether the models were pre-trained enough or contained knowledge correlated with linguistic theory.
• We are presenting the chronological probing study of transformer English models such as MultiBERT and T5.