Built on

Trust region policy optimization
John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz · 2015
Earlier work this paper cites.
Rl 2 : Fast reinforcement learning via slow reinforcement learning
Original
Yan Duan, John Schulman, Xi Chen, Peter L Bartlett, Ilya Sutskever, and Pieter Abbeel · 2016
Earlier work this paper cites.
Proximal policy optimization algorithms
Original
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov · 2017
Earlier work this paper cites.
Learning to generalize from sparse and underspecified rewards
Rishabh Agarwal, Chen Liang, Dale Schuurmans, and Mohammad Norouzi · 2019
Earlier work this paper cites.
Guided meta-policy search
Russell Mendonca, Abhishek Gupta, Rosen Kralev, Pieter Abbeel, Sergey Levine, and Chelsea Finn · 2019
Earlier work this paper cites.
Advantage-weighted regression: Simple and scalable off-policy reinforcement learning
Original
Xue Bin Peng, Aviral Kumar, Grace Zhang, and Sergey Levine · 2019
Earlier work this paper cites.
Efficient off-policy meta-reinforcement learning via probabilistic context variables
Kate Rakelly, Aurick Zhou, Chelsea Finn, Sergey Levine, and Deirdre Quillen · 2019
Earlier work this paper cites.
Some considerations on learning to explore via meta-reinforcement learning, 2019
Original
Bradly C. Stadie, Ge Yang, Rein Houthooft, Xi Chen, Yan Duan, Yuhuai Wu, Pieter Abbeel, and Ilya Sutskever · 2019
Earlier work this paper cites.
Meta reinforcement learning
Lilian Weng · 2019
Earlier work this paper cites.
Trl: Transformer reinforcement learning
Leandro von Werra, Younes Belkada, Lewis Tunstall, Edward Beeching, Tristan Thrush, Nathan Lambert, Shengyi Huang, Kashif Rasul, and Quentin Gallouédec · 2020
Earlier work this paper cites.
Training verifiers to solve math word problems
Original
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al · 2021
Earlier work this paper cites.
Why generalization in RL is difficult: Epistemic pomdps and implicit partial observability
Original
Dibya Ghosh, Jad Rahme, Aviral Kumar, Amy Zhang, Ryan P. Adams, and Sergey Levine · 2021
Earlier work this paper cites.
Scaling scaling laws with board games
Original
Andy L Jones · 2021
Earlier work this paper cites.
Star: Bootstrapping reasoning with reasoning
Eric Zelikman, Yuhuai Wu, Jesse Mu, and Noah Goodman · 2022
Earlier work this paper cites.
A survey of meta-reinforcement learning
Original
Jacob Beck, Risto Vuorio, Evan Zheran Liu, Zheng Xiong, Luisa Zintgraf, Chelsea Finn, and Shimon Whiteson · 2023
Earlier work this paper cites.
Beyond human data: Scaling self-training for problem-solving with language models
Original
Avi Singh, John D Co-Reyes, Rishabh Agarwal, Ankesh Anand, Piyush Patil, Xavier Garcia, Peter J Liu, James Harrison, Jaehoon Lee, Kelvin Xu, et al · 2023
Earlier work this paper cites.

Similar

Tree of thoughts: Deliberate problem solving with large language models
Original
Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Thomas L Griffiths, Yuan Cao, and Karthik Narasimhan · 2023
Cited alongside, same era.
Scaling test-time compute with open models, 2024
Edward Beeching, Lewis Tunstall, and Sasha Rush · 2024
Cited alongside, same era.
Do not think that much for 2+ 3=? on the overthinking of o1-like llms
Original
Xingyu Chen, Jiahao Xu, Tian Liang, Zhiwei He, Jianhui Pang, Dian Yu, Linfeng Song, Qiuzhi Liu, Mengfei Zhou, Zhuosheng Zhang, et al · 2024
Cited alongside, same era.
Inference-aware fine-tuning for best-of-n sampling in large language models
Original
Yinlam Chow, Guy Tennenholtz, Izzeddin Gur, Vincent Zhuang, Bo Dai, Sridhar Thiagarajan, Craig Boutilier, Rishabh Agarwal, Aviral Kumar, and Aleksandra Faust · 2024
Cited alongside, same era.
Stream of search (sos): Learning to search in language

Then

Scaling llm test-time compute optimally can be more effective than scaling model parameters
Original
Charlie Snell, Jaehoon Lee, Kelvin Xu, and Aviral Kumar · 2024
Later among the works it cites.
From decoding to meta-generation: Inference-time algorithms for large language models
Original
Sean Welleck, Amanda Bertsch, Matthew Finlayson, Hailey Schoelkopf, Alex Xie, Graham Neubig, Ilia Kulikov, and Zaid Harchaoui · 2024
Later among the works it cites.
Inference scaling laws: An empirical analysis of compute-optimal inference for problem-solving with language models
Original
Yangzhen Wu, Zhiqing Sun, Shanda Li, Sean Welleck, and Yiming Yang · 2024
Later among the works it cites.
Generative verifiers: Reward modeling as next-token prediction
Original
Lunjun Zhang, Arian Hosseini, Hritik Bansal, Mehran Kazemi, Aviral Kumar, and Rishabh Agarwal · 2024
Later among the works it cites.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…

2025

Optimizing Test-Time Compute via Meta Reinforcement Fine-Tuning

Qu, Yuxiao, Yang, Matthew Y. R., Setlur, Amrith et al.

Understand

Training models to effectively use test-time compute is crucial for improving the reasoning performance of LLMs.

Current methods mostly do so via fine-tuning on search traces or running RL with 0/1 outcome reward, but do these approaches efficiently utilize test-time compute? Would these approaches continue to scale as the budget improves? In this paper, we try to answer these questions.
We formalize the problem of optimizing test-time compute as a meta-reinforcement learning (RL) problem, which provides a principled perspective on spending test-time compute.
This perspective enables us to view the long output stream from the LLM as consisting of several episodes run at test time and leads us to use a notion of cumulative regret over output tokens as a way to measure the efficacy of test-time compute.

Built on

Trust region policy optimization
John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz · 2015
Earlier work this paper cites.
Rl 2 : Fast reinforcement learning via slow reinforcement learning
Original
Yan Duan, John Schulman, Xi Chen, Peter L Bartlett, Ilya Sutskever, and Pieter Abbeel · 2016
Earlier work this paper cites.
Proximal policy optimization algorithms
Original
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov · 2017
Earlier work this paper cites.
Learning to generalize from sparse and underspecified rewards
Rishabh Agarwal, Chen Liang, Dale Schuurmans, and Mohammad Norouzi · 2019
Earlier work this paper cites.
Guided meta-policy search
Russell Mendonca, Abhishek Gupta, Rosen Kralev, Pieter Abbeel, Sergey Levine, and Chelsea Finn · 2019
Earlier work this paper cites.
Advantage-weighted regression: Simple and scalable off-policy reinforcement learning
Original
Xue Bin Peng, Aviral Kumar, Grace Zhang, and Sergey Levine · 2019
Earlier work this paper cites.
Efficient off-policy meta-reinforcement learning via probabilistic context variables
Kate Rakelly, Aurick Zhou, Chelsea Finn, Sergey Levine, and Deirdre Quillen · 2019
Earlier work this paper cites.
Some considerations on learning to explore via meta-reinforcement learning, 2019
Original
Bradly C. Stadie, Ge Yang, Rein Houthooft, Xi Chen, Yan Duan, Yuhuai Wu, Pieter Abbeel, and Ilya Sutskever · 2019
Earlier work this paper cites.
Meta reinforcement learning
Lilian Weng · 2019
Earlier work this paper cites.
Trl: Transformer reinforcement learning
Leandro von Werra, Younes Belkada, Lewis Tunstall, Edward Beeching, Tristan Thrush, Nathan Lambert, Shengyi Huang, Kashif Rasul, and Quentin Gallouédec · 2020
Earlier work this paper cites.
Training verifiers to solve math word problems
Original
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al · 2021
Earlier work this paper cites.
Why generalization in RL is difficult: Epistemic pomdps and implicit partial observability
Original
Dibya Ghosh, Jad Rahme, Aviral Kumar, Amy Zhang, Ryan P. Adams, and Sergey Levine · 2021
Earlier work this paper cites.
Scaling scaling laws with board games
Original
Andy L Jones · 2021
Earlier work this paper cites.
Star: Bootstrapping reasoning with reasoning
Eric Zelikman, Yuhuai Wu, Jesse Mu, and Noah Goodman · 2022
Earlier work this paper cites.
A survey of meta-reinforcement learning
Original
Jacob Beck, Risto Vuorio, Evan Zheran Liu, Zheng Xiong, Luisa Zintgraf, Chelsea Finn, and Shimon Whiteson · 2023
Earlier work this paper cites.
Beyond human data: Scaling self-training for problem-solving with language models
Original
Avi Singh, John D Co-Reyes, Rishabh Agarwal, Ankesh Anand, Piyush Patil, Xavier Garcia, Peter J Liu, James Harrison, Jaehoon Lee, Kelvin Xu, et al · 2023
Earlier work this paper cites.

Similar

Tree of thoughts: Deliberate problem solving with large language models
Original
Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Thomas L Griffiths, Yuan Cao, and Karthik Narasimhan · 2023
Cited alongside, same era.
Scaling test-time compute with open models, 2024
Edward Beeching, Lewis Tunstall, and Sasha Rush · 2024
Cited alongside, same era.
Do not think that much for 2+ 3=? on the overthinking of o1-like llms
Original
Xingyu Chen, Jiahao Xu, Tian Liang, Zhiwei He, Jianhui Pang, Dian Yu, Linfeng Song, Qiuzhi Liu, Mengfei Zhou, Zhuosheng Zhang, et al · 2024
Cited alongside, same era.
Inference-aware fine-tuning for best-of-n sampling in large language models
Original
Yinlam Chow, Guy Tennenholtz, Izzeddin Gur, Vincent Zhuang, Bo Dai, Sridhar Thiagarajan, Craig Boutilier, Rishabh Agarwal, Aviral Kumar, and Aleksandra Faust · 2024
Cited alongside, same era.
Stream of search (sos): Learning to search in language

Then

Scaling llm test-time compute optimally can be more effective than scaling model parameters
Original
Charlie Snell, Jaehoon Lee, Kelvin Xu, and Aviral Kumar · 2024
Later among the works it cites.
From decoding to meta-generation: Inference-time algorithms for large language models
Original
Sean Welleck, Amanda Bertsch, Matthew Finlayson, Hailey Schoelkopf, Alex Xie, Graham Neubig, Ilia Kulikov, and Zaid Harchaoui · 2024
Later among the works it cites.
Inference scaling laws: An empirical analysis of compute-optimal inference for problem-solving with language models
Original
Yangzhen Wu, Zhiqing Sun, Shanda Li, Sean Welleck, and Yiming Yang · 2024
Later among the works it cites.
Generative verifiers: Reward modeling as next-token prediction
Original
Lunjun Zhang, Arian Hosseini, Hritik Bansal, Mehran Kazemi, Aviral Kumar, and Rishabh Agarwal · 2024
Later among the works it cites.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Original

Kanishk Gandhi, Denise Lee, Gabriel Grand, Muxin Liu, Winson Cheng, Archit Sharma, and Noah D Goodman · 2024

Cited alongside, same era.

Omni-math: A universal olympiad level mathematic benchmark for large language models

Original

Bofei Gao, Feifan Song, Zhe Yang, Zefan Cai, Yibo Miao, Qingxiu Dong, Lei Li, Chenghao Ma, Liang Chen, Runxin Xu, et al · 2024

Cited alongside, same era.

Unfamiliar finetuning examples control how language models hallucinate, 2024

Katie Kang, Eric Wallace, Claire Tomlin, Aviral Kumar, and Sergey Levine · 2024

Cited alongside, same era.

Vineppo: Unlocking rl potential for llm reasoning through refined credit assignment

Original

Amirhossein Kazemnejad, Milad Aghajohari, Eva Portelance, Alessandro Sordoni, Siva Reddy, Aaron Courville, and Nicolas Le Roux · 2024

Cited alongside, same era.

Training language models to self-correct via reinforcement learning

Original

Aviral Kumar, Vincent Zhuang, Rishabh Agarwal, Yi Su, John D Co-Reyes, Avi Singh, Kate Baumli, Shariq Iqbal, Colton Bishop, Rebecca Roelofs, et al · 2024

Cited alongside, same era.

Beyond a*: Better planning with transformers via search dynamics bootstrapping

Original

Lucas Lehnert, Sainbayar Sukhbaatar, DiJia Su, Qinqing Zheng, Paul Mcvay, Michael Rabbat, and Yuandong Tian · 2024

Cited alongside, same era.

Numinamath: The largest public dataset in ai4maths with 860k pairs of competition math problems and solutions

Jia Li, Edward Beeching, Lewis Tunstall, Ben Lipkin, Roman Soletskyi, Shengyi Huang, Kashif Rasul, Longhui Yu, Albert Q Jiang, Ziju Shen, et al · 2024

Cited alongside, same era.

Guided stream of search: Learning to better search with language models via optimal path guidance

Original

Seungyong Moon, Bumsoo Park, and Hyun Oh Song · 2024

Cited alongside, same era.

Training language models to reason efficiently, 2025

Original

Daman Arora and Andrea Zanette · 2025

Closest in time.

Process reinforcement through implicit rewards, 2025

Original

Ganqu Cui, Lifan Yuan, Zefan Wang, Hanbin Wang, Wendi Li, Bingxiang He, Yuchen Fan, Tianyu Yu, Qixin Xu, Weize Chen, Jiarui Yuan, Huayu Chen, Kaiyan Zhang, Xingtai Lv, Shuo Wang, Yuan Yao, Xu Han, Hao Peng, Yu Cheng, Zhiyuan Liu, Maosong Sun, Bowen Zhou, and Ning Ding · 2025

Closest in time.

Open r1: A fully open reproduction of deepseek-r1, January 2025

Hugging Face · 2025

Closest in time.

Kimi k1.5: Scaling reinforcement learning with llms, 2025

Kimi-Team · 2025

Closest in time.

There may not be aha moment in r1-zero-like training — a pilot study

Zichen Liu, Changyu Chen, Wenjun Li, Tianyu Pang, Chao Du, and Min Lin · 2025

Closest in time.

DeepScaleR: Surpassing O1-Preview with a 1.5B Model by Scaling RL

Michael Luo, Sijun Tan, Justin Wong, Xiaoxiang Shi, William Y. Tang, Manan Roongta, Colin Cai, Jeffrey Luo, Tianjun Zhang, Li Erran Li, Raluca Ada Popa, and Ion Stoica · 2025

Closest in time.

s1: Simple test-time scaling, 2025

Original

Niklas Muennighoff, Zitong Yang, Weijia Shi, Xiang Lisa Li, Li Fei-Fei, Hannaneh Hajishirzi, Luke Zettlemoyer, Percy Liang, Emmanuel Candès, and Tatsunori Hashimoto · 2025

Closest in time.

Optimizing llm test-time compute involves solving a meta-rl problem

Amrith Setlur, Yuxiao Qu, Matthew Yang, Lunjun Zhang, Virginia Smith, and Aviral Kumar · 2025

Closest in time.

Sky-t1: Train your own o1 preview model within $450

NovaSky Team · 2025

Closest in time.

Towards system 2 reasoning in llms: Learning how to think with meta chain-of-though

Original

Violet Xiang, Charlie Snell, Kanishk Gandhi, Alon Albalak, Anikait Singh, Chase Blagden, Duy Phung, Rafael Rafailov, Nathan Lile, Dakota Mahan, et al · 2025

Closest in time.

7b model and 8k examples: Emerging reasoning with reinforcement learning is both effective and efficient

Weihao Zeng, Yuzhen Huang, Wei Liu, Keqing He, Qian Liu, Zejun Ma, and Junxian He · 2025

Closest in time.