Neural Execution Engines: Learning to Execute Subroutines

Understand

A significant effort has been made to train neural networks that replicate algorithmic reasoning, but they often fail to learn the abstract concepts underlying these algorithms.

• This is evidenced by their inability to generalize to data distributions that are outside of their restricted training sets, namely larger inputs and unseen data.
• We study these generalization issues at the level of numerical subroutines that comprise common algorithms like sorting, shortest paths, and minimum spanning trees.
• First, we observe that transformer-based sequence-to-sequence models can learn subroutines like sorting a list of numbers, but their performance rapidly degrades as the length of lists grows beyond those found in the training set.