Hindcast: Replaying Prediction Markets to Evaluate LLM Forecasters
Merged summary
TL;DR — Hindcast is an evaluation framework that fixes data-leakage in backtesting LLM forecasters by grading models as if standing at a past date, replaying resolved Polymarket markets against a frozen pre-cutoff Reddit snapshot. It matters because standard backtests reward recall/lookup rather than genuine foresight.
- Identifies two leakage channels: retrieval surfacing post-event reports, and newer models being trained on data past the event date—both turn "forecasting" into recall.
- Closes leaks by fixing a per-market cutoff date $t_0$, letting models read only Reddit posts written before $t_0$, and scoring against both the actual outcome and the market's price at $t_0$ (a human forecast from the same information).
- Design stays fresh: per-market cutoffs plus an unchanging snapshot let the benchmark re-run on new markets as models improve, avoiding staleness.
- Finding: once leakage is closed, retrieval helps most models only when Reddit discussed the event beforehand; where the archive held mere speculation, retrieval hurts performance.
Sources (1)
Hindcast: Replaying Prediction Markets to Evaluate LLM Forecasters
TL;DR — Hindcast is an evaluation framework that fixes data-leakage in backtesting LLM forecasters by grading models as if standing at a past date, replaying resolved Polymarket markets against a frozen pre-cutoff Reddit snapshot. It matters because standard backtests reward recall/lookup rather than genuine foresight.
- Identifies two leakage channels: retrieval surfacing post-event reports, and newer models being trained on data past the event date—both turn "forecasting" into recall.
- Closes leaks by fixing a per-market cutoff date $t_0$, letting models read only Reddit posts written before $t_0$, and scoring against both the actual outcome and the market's price at $t_0$ (a human forecast from the same information).
- Design stays fresh: per-market cutoffs plus an unchanging snapshot let the benchmark re-run on new markets as models improve, avoiding staleness.
- Finding: once leakage is closed, retrieval helps most models only when Reddit discussed the event beforehand; where the archive held mere speculation, retrieval hurts performance.