Just Ask Jev: Reinforcement Learning for Calibrated Decisions as a Zero-Shot Detector of AI Alignment Failures

TL;DR - RLCDAlignBench evaluates Jev as a zero-shot detector across ten types of AI alignment failures. A single generic question achieves a median AUROC of 0.886 while costing 63× less than LLM-judge scoring.

• The benchmark covers 44 datasets and five target models, including failures such as jailbreaks, deception, hallucination, prompt injection, and reward hacking.
• Jev answers multiple typed questions about an input with calibrated probabilities in one call, unlike generative judges that decode separately for each criterion.
• Input context matters more than question wording, particularly when supplied fields encode information relevant to the label.
• Jev beats supervised baselines on most benchmarks, matches reference scorers’ agreement with human labels, and exposes labeling defects in existing evaluations.