Fetching the paper…

Reading the bibliography…

2025

Bench to the Future: A Pastcasting Benchmark for Forecasting Agents

FutureSearch, :, Wildman, Jack et al.

Original
PDF
Post on X

Understand

Forecasting is a challenging task that offers a clearly measurable way to study AI systems.

Forecasting requires a large amount of research on the internet, and evaluations require time for events to happen, making the development of forecasting benchmarks challenging.
To date, no forecasting benchmark provides a realistic, hermetic, and repeatable environment for LLM forecasters.
We introduce Bench To the Future (BTF), a "pastcasting" benchmark with hundreds of high-quality questions for which the resolution is already known.

Bench to the Future: A Pastcasting Benchmark for Forecasting Agents · Around

Built on

“Verification of Forecasts Expressed in Terms of Probability”
Glenn. Brier · 1950
Earlier work this paper cites.
“Comparing Two Forecasters in an Ideal World”, Metaculus Notebook, 2023
Nikos. Bosse · 2023
Earlier work this paper cites.
“ReAct: Synergizing Reasoning and Acting in Language Models”, 2023
Original
Shunyu Yao et al · 2023
Earlier work this paper cites.
“Approaching Human-Level Forecasting with Language Models”
Original
Danny Halawi, Fred Zhang, Chen Yueh-Han and Jacob Steinhardt · 2024
Earlier work this paper cites.

Similar

“Wisdom of the Silicon Crowd: LLM Ensemble Prediction Capabilities Rival Human Crowd Accuracy”
Original
Philipp Schoenegger, Indre Tuminauskaite, Peter. Park and Philip. Tetlock · 2024
Cited alongside, same era.
“LLMs Are Superhuman Forecasters”, 2024
Liam Phan, Aamir Khoja, Mantas Mazeika and Dan Hendrycks · 2024
Cited alongside, same era.
“Reasoning and Tools for Human-Level Forecasting”
Original
Elvis Hsieh, Preston Fu and Jonathan Chen · 2024
Cited alongside, same era.
“Metaculus AI Forecasting Benchmark Series”, Quarterly forecasting tournaments, 2024
Metaculus · 2024
Cited alongside, same era.

Then

“ForecastBench: A Dynamic Benchmark of AI Forecasting Capabilities”
Original
Ezra Karger et al · 2025
Closest in time.
“Deep Research Bench: Evaluating AI Web Research Agents”, 2025
Original
FutureSearch et al · 2025
Closest in time.
“Metaculus Forecasting Platform”, Forecasting platform, 2025
Metaculus · 2025
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…