CalibratedRubric: Task-Adaptive Rubric Banks for Open-Ended LLM Evaluation
Ranking
Overall
68
Content
80
Popularity
40
Observed public metrics from 1 member.
Merged summary
TL;DR - CalibratedRubric is a task-adaptive framework that automatically builds compact, calibrated rubric banks for grading open-ended LLM outputs, replacing costly expert curation and crude unanimity/binary-variance filters. It matters because it raises agreement with human gold labels while cutting the number of rubrics needed.
- Combines three pieces: type-specific scoring, Bayesian rubric-measurability filtering via a Beta–Bernoulli agreement posterior, and item response theory (IRT)-based bank assembly using a submodular information-coverage objective.
- Measurability filtering lifts human-gold agreement on JudgmentBench from Îş=0.604 to Îş=0.743.
- IRT-based greedy selection beats random selection on cross-fitted rank fidelity across all six response blocks, and needs only 49 rubrics instead of 131 to hit the target correlation on FinResearchBench decision-support tasks.
- Evaluated across financial, healthcare, general, and legal benchmarks; task-label perturbations degrade system separation, and calibration gains depend on sufficient judge redundancy.
Sources (1)
CalibratedRubric: Task-Adaptive Rubric Banks for Open-Ended LLM Evaluation
Public signals
Semantic Scholar citations 0 · Semantic Scholar influential citations 0
TL;DR - CalibratedRubric is a task-adaptive framework that automatically builds compact, calibrated rubric banks for grading open-ended LLM outputs, replacing costly expert curation and crude unanimity/binary-variance filters. It matters because it raises agreement with human gold labels while cutting the number of rubrics needed.
- Combines three pieces: type-specific scoring, Bayesian rubric-measurability filtering via a Beta–Bernoulli agreement posterior, and item response theory (IRT)-based bank assembly using a submodular information-coverage objective.
- Measurability filtering lifts human-gold agreement on JudgmentBench from Îş=0.604 to Îş=0.743.
- IRT-based greedy selection beats random selection on cross-fitted rank fidelity across all six response blocks, and needs only 49 rubrics instead of 131 to hit the target correlation on FinResearchBench decision-support tasks.
- Evaluated across financial, healthcare, general, and legal benchmarks; task-label perturbations degrade system separation, and calibration gains depend on sufficient judge redundancy.