JudgeStealer: Extracting LLM Judging Capabilities across Evaluation Protocols

TL;DR - JudgeStealer is a query-efficient model-extraction framework that replicates black-box LLM judges across pointwise, pairwise, and listwise evaluation protocols. It highlights the vulnerability of proprietary judging capabilities even under restricted query budgets and representative defenses.

• Converts queried pointwise scores into pairwise and listwise supervision without additional victim-model queries by exploiting cross-protocol agreement.
• Selects informative queries using semantic diversity, predictive uncertainty, and potential judge biases.
• Uses score smoothing and multi-protocol review to preserve score ordering and reduce catastrophic forgetting during surrogate adaptation.
• Reaches up to 73.3% pointwise, 87.0% pairwise, and 71.6% listwise accuracy, outperforming existing extraction baselines across tested model scales and settings.