How to Speculate about Uncertainty in Agentic Coding? A Draft-Model Gate Method

TL;DR - Speculative Uncertainty uses a small open-weight draft model to predict black-box coding-agent failures from generated tokens alone. Its pre-execution gate reduced execution errors and token costs across multiple agents and out-of-distribution benchmarks.

• Scores completed agent trajectories in one forward pass without logits, model weights, activations, or repeated sampling.
• Separates reasoning and action spans to extract phase-aware cross-likelihood features calibrated against verifiable outcomes.
• Produces failure-likelihood scores usable for action vetoes, human escalation, routing, or additional test-time compute.
• Reduced execution error rates by 6–8 percentage points and token costs by 14–19% for Qwen3-Coder-480B and Claude 3.5 Sonnet.