Stopping and Routing LLM Judge Panels

TL;DR - This paper frames LLM judge-panel design as a cost-aware allocation problem that determines which evaluators to call, for which examples, and when to stop adding judges. It aims to produce reusable, auditable evaluation plans from a small labeled audit set.

• Classifies judges by target-relative roles: redundant copies, globally useful complements, and slice-specific specialists.
• Drops copies, adds complements globally, and conditionally routes specialists to declared, deployable slices.
• Stops panel expansion when validation gains fall below a threshold, balancing evaluation risk against judge-call costs.
• Evaluates the approach across reasoning, code, safety, preference, reward-model, summarization, and math audits against several panel and cascade baselines.