CalibratedRubric: Task-Adaptive Rubric Banks for Open-Ended LLM Evaluation

TL;DR - CalibratedRubric is a task-adaptive framework that automatically builds compact, calibrated rubric banks for grading open-ended LLM outputs, replacing costly expert curation and crude unanimity/binary-variance filters. It matters because it raises agreement with human gold labels while cutting the number of rubrics needed.

• Combines three pieces: type-specific scoring, Bayesian rubric-measurability filtering via a Beta–Bernoulli agreement posterior, and item response theory (IRT)-based bank assembly using a submodular information-coverage objective.
• Measurability filtering lifts human-gold agreement on JudgmentBench from κ=0.604 to κ=0.743.
• IRT-based greedy selection beats random selection on cross-fitted rank fidelity across all six response blocks, and needs only 49 rubrics instead of 131 to hit the target correlation on FinResearchBench decision-support tasks.
• Evaluated across financial, healthcare, general, and legal benchmarks; task-label perturbations degrade system separation, and calibration gains depend on sufficient judge redundancy.