Does syntax need to grow on trees? Sources of hierarchical inductive bias in sequence-to-sequence networks

Understand

Learners that are exposed to the same training data might generalize differently due to differing inductive biases.

• In neural network models, inductive biases could in theory arise from any aspect of the model architecture.
• We investigate which architectural factors affect the generalization behavior of neural sequence-to-sequence models trained on two syntactic tasks, English question formation and English tense reinflection.
• For both tasks, the training set is consistent with a generalization based on hierarchical structure and a generalization based on linear order.