Fetching the paper…

Reading the bibliography…

2025

Optimizing Temperature for Language Models with Multi-Sample Inference

Du, Weihua, Yang, Yiming, Welleck, Sean

Original
PDF
Post on X

Understand

Multi-sample aggregation strategies, such as majority voting and best-of-N sampling, are widely used in contemporary large language models (LLMs) to enhance predictive accuracy across various tasks.

A key challenge in this process is temperature selection, which significantly impacts model performance.
Existing approaches either rely on a fixed default temperature or require labeled validation data for tuning, which are often scarce and difficult to obtain.
This paper addresses the challenge of automatically identifying the (near)-optimal temperature for different LLMs using multi-sample aggregation strategies, without relying on task-specific validation data.

Optimizing Temperature for Language Models with Multi-Sample Inference · Around

Built on

The curious case of neural text degeneration
Original
Holtzman, A., Buys, J., Du, L., Forbes, M., and Choi, Y · 2019
Earlier work this paper cites.
Stochastic beams and where to find them: The gumbel-top-k trick for sampling sequences without replacement
Kool, W., Van Hoof, H., and Welling, M · 2019
Earlier work this paper cites.
Calibration of encoder decoder models for neural machine translation
Original
Kumar, A. and Sarawagi, S · 2019
Earlier work this paper cites.
Program synthesis with large language models
Original
Austin, J., Odena, A., Nye, M., Bosma, M., Michalewski, H., Dohan, D., Jiang, E., Cai, C., Terry, M., Le, Q., et al · 2021
Earlier work this paper cites.
Evaluating large language models trained on code
Original
Chen, M., Tworek, J., Jun, H., Yuan, Q., Pinto, H. P. D. O., Kaplan, J., Edwards, H., Burda, Y., Joseph, N., Brockman, G., et al · 2021
Earlier work this paper cites.
Measuring mathematical problem solving with the math dataset
Original
Hendrycks, D., Burns, C., Kadavath, S., Arora, A., Basart, S., Tang, E., Song, D., and Steinhardt, J · 2021
Earlier work this paper cites.
A framework for the evaluation of code generation models
Ben Allal, L., Muennighoff, N., Kumar Umapathi, L., Lipkin, B., and von Werra, L · 2022
Earlier work this paper cites.
Self-consistency improves chain of thought reasoning in language models
Original
Wang, X., Wei, J., Schuurmans, D., Le, Q., Chi, E., Narang, S., Chowdhery, A., and Zhou, D · 2022
Earlier work this paper cites.
Chain-of-thought prompting elicits reasoning in large language models
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Xia, F., Chi, E., Le, Q. V., Zhou, D., et al · 2022
Earlier work this paper cites.
Llemma: An open language model for mathematics
Original
Azerbayev, Z., Schoelkopf, H., Paster, K., Santos, M. D., McAleer, S., Jiang, A. Q., Deng, J., Biderman, S., and Welleck, S · 2023
Earlier work this paper cites.
Mistral 7b
Original
Jiang, A. Q., Sablayrolles, A., Mensch, A., Bamford, C., Chaplot, D. S., Casas, D. d. l., Bressand, F., Lengyel, G., Lample, G., Saulnier, L., et al · 2023
Earlier work this paper cites.
Evaluating open-domain question answering in the era of large language models
Original
Kamalloo, E., Dziri, N., Clarke, C. L., and Rafiei, D · 2023
Earlier work this paper cites.
Efficient memory management for large language model serving with pagedattention
Kwon, W., Li, Z., Zhuang, S., Sheng, Y., Zheng, L., Yu, C. H., Gonzalez, J. E., Zhang, H., and Stoica, I · 2023
Earlier work this paper cites.

Similar

Locally typical sampling
Meister, C., Pimentel, T., Wiher, G., and Cotterell, R · 2023
Cited alongside, same era.
Code llama: Open foundation models for code
Original
Roziere, B., Gehring, J., Gloeckle, F., Sootla, S., Gat, I., Tan, X. E., Adi, Y., Liu, J., Sauvestre, R., Remez, T., et al · 2023
Cited alongside, same era.
Building cooperative embodied agents modularly with large language models
Original
Zhang, H., Du, W., Shan, J., Zhou, Q., Du, Y., Tenenbaum, J. B., Shu, T., and Gan, C · 2023
Cited alongside, same era.
Meet yi-coder: A small but mighty llm for code, September 2024
01.AI · 2024
Cited alongside, same era.
Large language models for mathematical reasoning: Progresses and challenges
Original
Ahn, J., Verma, R., Lou, R., Liu, D., Zhang, R., and Yin, W · 2024
Cited alongside, same era.

Then

Llm and simulation as bilevel optimizers: A new paradigm to advance physical scientific discovery
Original
Ma, P., Wang, T.-H., Guo, M., Sun, Z., Tenenbaum, J. B., Rus, D., Gan, C., and Matusik, W · 2024
Later among the works it cites.
Turning up the heat: Min-p sampling for creative and coherent llm outputs
Original
Nguyen, M., Baker, A., Neo, C., Roush, A., Kirsch, A., and Shwartz-Ziv, R · 2024
Later among the works it cites.
The effect of sampling temperature on problem solving in large language models
Original
Renze, M. and Guven, E · 2024
Later among the works it cites.
Mathematical discoveries from program search with large language models
Romera-Paredes, B., Barekatain, M., Novikov, A., Balog, M., Kumar, M. P., Dupont, E., Ruiz, F. J., Ellenberg, J. S., Wang, P., Fawzi, O., et al · 2024
Later among the works it cites.
Deepseekmath: Pushing the limits of mathematical reasoning in open language models

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Adaptive decoding via latent preference optimization

Dhuliawala, S., Kulikov, I., Yu, P., Celikyilmaz, A., Weston, J., Sukhbaatar, S., and Lanchantin, J · 2024

Cited alongside, same era.

The llama 3 herd of models

Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Yang, A., Fan, A., et al · 2024

Cited alongside, same era.

Deepseek-coder: When the large language model meets programming–the rise of code intelligence

Guo, D., Zhu, Q., Yang, D., Xie, Z., Dong, K., Zhang, W., Chen, G., Bi, X., Wu, Y., Li, Y., et al · 2024

Cited alongside, same era.

Qwen2. 5-coder technical report

Hui, B., Yang, J., Cui, Z., Yang, J., Liu, D., Zhang, L., Liu, T., Zhang, J., Yu, B., Lu, K., et al · 2024

Cited alongside, same era.

Openai o1 system card

Jaech, A., Kalai, A., Lerer, A., Richardson, A., El-Kishky, A., Low, A., Helyar, A., Madry, A., Beutel, A., Carney, A., et al · 2024

Cited alongside, same era.

Dynamic stochastic decoding strategy for open-domain dialogue generation

Li, Y., Mi, F., Li, Y., Wang, Y., Sun, B., Feng, S., and Li, K · 2024

Cited alongside, same era.

Lean-star: Learning to interleave thinking and proving

Lin, H., Sun, Z., Yang, Y., and Welleck, S · 2024

Cited alongside, same era.

Shao, Z., Wang, P., Zhu, Q., Xu, R., Song, J., Bi, X., Zhang, H., Zhang, M., Li, Y., Wu, Y., et al · 2024

Later among the works it cites.

Easy-to-hard generalization: Scalable alignment beyond human supervision

Sun, Z., Yu, L., Shen, Y., Liu, W., Yang, Y., Welleck, S., and Gan, C · 2024

Later among the works it cites.

Openmathinstruct-2: Accelerating ai for math with massive open-source instruction data

Toshniwal, S., Du, W., Moshkov, I., Kisacanin, B., Ayrapetyan, A., and Gitman, I · 2024

Later among the works it cites.

From decoding to meta-generation: Inference-time algorithms for large language models

Welleck, S., Bertsch, A., Finlayson, M., Schoelkopf, H., Xie, A., Neubig, G., Kulikov, I., and Harchaoui, Z · 2024

Later among the works it cites.

Inference scaling laws: An empirical analysis of compute-optimal inference for problem-solving with language models

Original

Wu, Y., Sun, Z., Li, S., Welleck, S., and Yang, Y · 2024

Later among the works it cites.

Calibrating language models with adaptive temperature scaling

Xie, J., Chen, A. S., Lee, Y., Mitchell, E., and Finn, C · 2024

Later among the works it cites.

Tree of thoughts: Deliberate problem solving with large language models

Yao, S., Yu, D., Zhao, J., Shafran, I., Griffiths, T., Cao, Y., and Narasimhan, K · 2024

Later among the works it cites.

Edt: Improving large language models’ generation by entropy-based dynamic temperature sampling

Zhang, S., Bao, Y., and Huang, S · 2024

Later among the works it cites.