Single Headed Attention RNN: Stop Thinking With Your Head

Understand

The leading approaches in language modeling are all obsessed with TV shows of my youth - namely Transformers and Sesame Street.

• Transformers this, Transformers that, and over here a bonfire worth of GPU-TPU-neuromorphic wafer scale silicon.
• We opt for the lazy path of old and proven techniques with a fancy crypto inspired acronym: the Single Headed Attention RNN (SHA-RNN).
• The author's lone goal is to show that the entire field might have evolved a different direction if we had instead been obsessed with a slightly different acronym and slightly different result.