Who Thinks Best Depends on How Long You Let Them: Budget-Dependent Rankings in LLM Evaluation
Ranking
Overall
84
Content
95
Popularity
58
Observed public metrics from 1 member.
Merged summary
TL;DR - LLM rankings can change or reverse as generation-token budgets vary, undermining evaluations performed at a single inference budget. Budget-conditioned evaluation and model routing may better reflect accuracy, efficiency, and model complementarity.
- Across 56,476 inferences, 3–19% of items became less accurate with larger budgets.
- Model rankings reversed across budgets on all three reasoning benchmarks.
- Oracle model selection improved performance by up to 27.8 percentage points, especially under constrained budgets.
- A budget-aware router captured 14.1% of the cross-domain oracle gap, but budget features transferred poorly between domains.
Sources (1)
Who Thinks Best Depends on How Long You Let Them: Budget-Dependent Rankings in LLM Evaluation
Public signals
Semantic Scholar citations 1 · Semantic Scholar influential citations 0
TL;DR - LLM rankings can change or reverse as generation-token budgets vary, undermining evaluations performed at a single inference budget. Budget-conditioned evaluation and model routing may better reflect accuracy, efficiency, and model complementarity.
- Across 56,476 inferences, 3–19% of items became less accurate with larger budgets.
- Model rankings reversed across budgets on all three reasoning benchmarks.
- Oracle model selection improved performance by up to 27.8 percentage points, especially under constrained budgets.
- A budget-aware router captured 14.1% of the cross-domain oracle gap, but budget features transferred poorly between domains.