You Only Cache Once: Decoder-Decoder Architectures for Language Models

Understand

We introduce a decoder-decoder architecture, YOCO, for large language models, which only caches key-value pairs once.

• It consists of two components, i.e., a cross-decoder stacked upon a self-decoder.
• The self-decoder efficiently encodes global key-value (KV) caches that are reused by the cross-decoder via cross-attention.
• The overall model behaves like a decoder-only Transformer, although YOCO only caches once.