Hierarchical Attention Encoder Decoder

Understand

Recent advances in large language models have shown that autoregressive modeling can generate complex and novel sequences that have many real-world applications.

• However, these models must generate outputs autoregressively, which becomes time-consuming when dealing with long sequences.
• Hierarchical autoregressive approaches that compress data have been proposed as a solution, but these methods still generate outputs at the original data frequency, resulting in slow and memory-intensive models.
• In this paper, we propose a model based on the Hierarchical Recurrent Encoder Decoder (HRED) architecture.