Learning How Hard to Think: Input-Adaptive Allocation of LM Computation

Understand

Computationally intensive decoding procedures--including search, reranking, and self-critique--can improve the quality of language model (LM) outputs in problems spanning code generation, numerical reasoning, and dialog.

• Existing work typically applies the same decoding procedure for every input to an LM.
• But not all inputs require the same amount of computation to process.
• Can we allocate decoding computation adaptively, using more resources to answer questions whose answers will be harder to compute? We present an approach that predicts the distribution of rewards given an input and computation budget, then allocates additional computation to inputs for which it is predicted to be most useful.