Evaluating Confidence-Gated Retrieval with Matched Trajectory Replay
Ranking
Overall
84
Content
90
Popularity
71
Observed public metrics from 1 member.
Merged summary
TL;DR - Matched trajectory replay evaluates how confidence calibration changes retrieval and answer decisions while holding agent trajectories and costs fixed. Calibration can reduce commitment risk, but it does not predict whether further retrieval will help, requiring a separate value-of-information estimate.
- Isotonic calibration increased accuracy among committed answers by up to 41 percentage points across six model-dataset pairs, often at the cost of lower coverage and more retrieval.
- Overall accuracy rose by up to 15 points on HotpotQA but fell by up to 17 points on MuSiQue, reflecting a more selective operating point rather than better answers or confidence rankings.
- Pre-retrieval calibration generalized through retrieval depths one and two, but underperformed raw confidence at depth three for all three tested model families.
- Agent evaluations should jointly report held-out calibration, risk-coverage tradeoffs, and retrieval cost.
Sources (1)
Evaluating Confidence-Gated Retrieval with Matched Trajectory Replay
Public signals
Semantic Scholar citations 1 · Semantic Scholar influential citations 0
TL;DR - Matched trajectory replay evaluates how confidence calibration changes retrieval and answer decisions while holding agent trajectories and costs fixed. Calibration can reduce commitment risk, but it does not predict whether further retrieval will help, requiring a separate value-of-information estimate.
- Isotonic calibration increased accuracy among committed answers by up to 41 percentage points across six model-dataset pairs, often at the cost of lower coverage and more retrieval.
- Overall accuracy rose by up to 15 points on HotpotQA but fell by up to 17 points on MuSiQue, reflecting a more selective operating point rather than better answers or confidence rankings.
- Pre-retrieval calibration generalized through retrieval depths one and two, but underperformed raw confidence at depth three for all three tested model families.
- Agent evaluations should jointly report held-out calibration, risk-coverage tradeoffs, and retrieval cost.