CalibForge: Adversarial Solver Calibration for Scaling Learnable Terminal Tasks

TL;DR - CalibForge is an autonomous system that synthesizes terminal-agent training tasks and calibrates their difficulty against verified solver behavior, producing 5,431 tasks that yield large gains on agentic coding benchmarks. It matters because it targets task learnability relative to a solver, not just executability, as a data-construction principle.

• Two calibration strategies define a "solver-relative learnable zone": multi-solver calibration exploits disagreement across a heterogeneous solver pool, while contrastive calibration enforces a strong-pass/weak-fail relation, both anchored in demonstrated solvability.
• Ablations report both strategies outperform authoring-plus-executable-validation alone and ordinary single-solver feedback.
• Models trained on the full 5,431-task collection reach 32.58% and 47.57% on Terminal-Bench 2.0.
• Largest reported gains over the corresponding base model: +24.71 points on Terminal-Bench 2.0, +27.68 on SWE-bench Pro, and +30.04 on Doc2Repo, suggesting transfer beyond the terminal domain.