Cheap Verifiers, Large Blind Spots: Measuring the Reliability Cost of Cost-Saving Cascades

TL;DR - This paper finds that cost-saving LLM cascades can hide severe reliability failures because the verifier used to route and evaluate answers has substantial blind spots. Fine-tuning students on verifier rejections may worsen or collapse performance while internal metrics remain deceptively stable.

• Verifier blind-spot rates increased from 0.12 to 0.55 as student size scaled from 0.5B to 32B, but decreased with stronger verifiers.
• A frontier verifier reduced the blind-spot rate to about 0.05 but escalated 46% of hard-MATH queries, eroding the cascade’s cost advantage.
• Corrective fine-tuning on rejected answers degraded the small student across both same-family and cross-family teachers.
• Verifier-based monitoring reported a steady 3% error even as true delivered error reached 32%, showing that in-loop metrics cannot reliably detect degradation.