Knowing When to Stop: Bayesian Optimal Stopping for LLM Evaluations

TL;DR - Optstop is a Bayesian adaptive-stopping framework that allocates LLM evaluation trials according to uncertainty rather than fixed repetition counts. It can substantially reduce evaluation compute while preserving overall conclusions.

• Uses hierarchical Bayesian inference for binary, ordinal, and continuous outcomes.
• Stops sampling items once estimates are sufficiently precise or stable.
• Keeps all benchmark items eligible and requires no calibrated item bank.
• Removed 57%–97% of planned trials in an illustrative 200-item evaluation across nine validation settings.