CTL++: Evaluating Generalization on Never-Seen Compositional Patterns of Known Functions, and Compatibility of Neural Representations

Understand

Well-designed diagnostic tasks have played a key role in studying the failure of neural nets (NNs) to generalize systematically.

• Famous examples include SCAN and Compositional Table Lookup (CTL).
• Here we introduce CTL++, a new diagnostic dataset based on compositions of unary symbolic functions.
• While the original CTL is used to test length generalization or productivity, CTL++ is designed to test systematicity of NNs, that is, their capability to generalize to unseen compositions of known functions.