Addressing Some Limitations of Transformers with Feedback Memory

Understand

Transformers have been successfully applied to sequential, auto-regressive tasks despite being feedforward networks.

• Unlike recurrent neural networks, Transformers use attention to capture temporal relations while processing input tokens in parallel.
• While this parallelization makes them computationally efficient, it restricts the model from fully exploiting the sequential nature of the input.
• The representation at a given layer can only access representations from lower layers, rather than the higher level representations already available.