Stopping and Routing LLM Judge Panels
Ranking
Overall
82
Content
100
Popularity
39
Observed public metrics from 1 member.
Merged summary
TL;DR - This paper frames LLM judge-panel design as a cost-aware allocation problem that determines which evaluators to call, for which examples, and when to stop adding judges. It aims to produce reusable, auditable evaluation plans from a small labeled audit set.
- Classifies judges by target-relative roles: redundant copies, globally useful complements, and slice-specific specialists.
- Drops copies, adds complements globally, and conditionally routes specialists to declared, deployable slices.
- Stops panel expansion when validation gains fall below a threshold, balancing evaluation risk against judge-call costs.
- Evaluates the approach across reasoning, code, safety, preference, reward-model, summarization, and math audits against several panel and cascade baselines.
Sources (1)
Stopping and Routing LLM Judge Panels
Public signals
Semantic Scholar citations 0 · Semantic Scholar influential citations 0
TL;DR - This paper frames LLM judge-panel design as a cost-aware allocation problem that determines which evaluators to call, for which examples, and when to stop adding judges. It aims to produce reusable, auditable evaluation plans from a small labeled audit set.
- Classifies judges by target-relative roles: redundant copies, globally useful complements, and slice-specific specialists.
- Drops copies, adds complements globally, and conditionally routes specialists to declared, deployable slices.
- Stops panel expansion when validation gains fall below a threshold, balancing evaluation risk against judge-call costs.
- Evaluates the approach across reasoning, code, safety, preference, reward-model, summarization, and math audits against several panel and cascade baselines.