How to Speculate about Uncertainty in Agentic Coding? A Draft-Model Gate Method
Ranking
Overall
78
Content
95
Popularity
37
Observed public metrics from 1 member.
Merged summary
TL;DR - Speculative Uncertainty uses a small open-weight draft model to predict black-box coding-agent failures from generated tokens alone. Its pre-execution gate reduced execution errors and token costs across multiple agents and out-of-distribution benchmarks.
- Scores completed agent trajectories in one forward pass without logits, model weights, activations, or repeated sampling.
- Separates reasoning and action spans to extract phase-aware cross-likelihood features calibrated against verifiable outcomes.
- Produces failure-likelihood scores usable for action vetoes, human escalation, routing, or additional test-time compute.
- Reduced execution error rates by 6–8 percentage points and token costs by 14–19% for Qwen3-Coder-480B and Claude 3.5 Sonnet.
Sources (1)
How to Speculate about Uncertainty in Agentic Coding? A Draft-Model Gate Method
Public signals
Semantic Scholar citations 0 · Semantic Scholar influential citations 0
TL;DR - Speculative Uncertainty uses a small open-weight draft model to predict black-box coding-agent failures from generated tokens alone. Its pre-execution gate reduced execution errors and token costs across multiple agents and out-of-distribution benchmarks.
- Scores completed agent trajectories in one forward pass without logits, model weights, activations, or repeated sampling.
- Separates reasoning and action spans to extract phase-aware cross-likelihood features calibrated against verifiable outcomes.
- Produces failure-likelihood scores usable for action vetoes, human escalation, routing, or additional test-time compute.
- Reduced execution error rates by 6–8 percentage points and token costs by 14–19% for Qwen3-Coder-480B and Claude 3.5 Sonnet.