Judge, Retrieve, or Abstain: Uncertainty-Guarded LLM Judging with Provable Risk Guarantees

TL;DR - This paper introduces an uncertainty-aware LLM judging framework that chooses between parametric evaluation, web retrieval, and abstention. Calibrated thresholds provide finite-sample guarantees on the false discovery rate of accepted verdicts while improving coverage over single-mode baselines.

• Calibrates uncertainty thresholds on held-out data using Clopper–Pearson confidence intervals.
• Routes low-confidence parametric judgments to a retrieval-augmented judge for evidence-based reevaluation.
• Extends the risk guarantee to two-threshold routing without additional assumptions.
• Maintains target error rates across open-domain QA benchmarks and multiple judge scales while achieving substantially higher coverage than single-mode methods.