Predicting Task Difficulty Without Rollouts

TL;DR - An arXiv study on predicting agentic task difficulty ex ante — directly from a task description, before running expensive rollouts — evaluated across 17 agentic benchmarks. It matters because trial-and-error evaluation is a major compute bottleneck for long-horizon agents, and reliable forecasts would let designers calibrate benchmarks and build progressive training curricula.

• Scope spans 17 agentic benchmarks covering coding, mathematics, machine learning, web navigation, and function calling, going beyond prior work limited to static tasks or isolated coding environments.
• Warns that AUC as an evaluation metric can mask poor difficulty estimates, arguing prior work relied on inaccurate metrics and narrow features.
• Identifies token-level entropy as a useful predictive signal for forecasting success likelihood.
• Residuals between expected and observed difficulty surface hidden environment flaws such as data contamination and infeasible tasks.