One Human, $N$ Agents: Audit-Budget Allocation for LLM Agent Fleets under Miscalibrated, Correlated Confidence
Ranking
Overall
78
Content
95
Popularity
39
Observed public metrics from 1 member.
Merged summary
TL;DR - This paper studies how one human should allocate limited audits across many LLM agents when self-reported confidence is miscalibrated and errors are correlated. It identifies when confidence-ranked auditing becomes worse than random selection, exposing conditions under which oversight is effectively vacuous.
- Models budgeted noisy inspection using a two-level Gaussian copula and derives a miscalibration threshold (δ^*).
- Counterintuitively, (δ^*) increases as audit budgets shrink, while shared task difficulty drives substantial cross-family error correlation.
- Five open-weight models exhibit near-constant, operationally unhelpful confidence; a proprietary model remains informative and below the estimated threshold.
- Policy replays on recorded traces confirm the predicted ordering of auditing strategies.
Sources (1)
One Human, $N$ Agents: Audit-Budget Allocation for LLM Agent Fleets under Miscalibrated, Correlated Confidence
Public signals
Semantic Scholar citations 0 · Semantic Scholar influential citations 0
TL;DR - This paper studies how one human should allocate limited audits across many LLM agents when self-reported confidence is miscalibrated and errors are correlated. It identifies when confidence-ranked auditing becomes worse than random selection, exposing conditions under which oversight is effectively vacuous.
- Models budgeted noisy inspection using a two-level Gaussian copula and derives a miscalibration threshold (δ^*).
- Counterintuitively, (δ^*) increases as audit budgets shrink, while shared task difficulty drives substantial cross-family error correlation.
- Five open-weight models exhibit near-constant, operationally unhelpful confidence; a proprietary model remains informative and below the estimated threshold.
- Policy replays on recorded traces confirm the predicted ordering of auditing strategies.