Minimal Width for Universal Property of Deep RNN

Understand

A recurrent neural network (RNN) is a widely used deep-learning network for dealing with sequential data.

• Imitating a dynamical system, an infinite-width RNN can approximate any open dynamical system in a compact domain.
• In general, deep networks with bounded widths are more effective than wide networks in practice; however, the universal approximation theorem for deep narrow structures has yet to be extensively studied.
• In this study, we prove the universality of deep narrow RNNs and show that the upper bound of the minimum width for universality can be independent of the length of the data.