Superposed Decoding: Multiple Generations from a Single Autoregressive Inference Pass

Understand

Many applications today provide users with multiple auto-complete drafts as they type, including GitHub's code completion, Gmail's smart compose, and Apple's messaging auto-suggestions.

• Under the hood, language models support this by running an autoregressive inference pass to provide a draft.
• Consequently, providing $k$ drafts to the user requires running an expensive language model $k$ times.
• To alleviate the computation cost of running $k$ inference passes, we propose Superposed Decoding, a new decoding algorithm that generates $k$ drafts at the computation cost of one autoregressive inference pass.