Sound Probabilistic Safety Bounds for Large Language Models

TL;DR - This paper introduces a framework for computing formally sound probability bounds on harmful LLM outputs. It enables statistical safety certification, including useful lower bounds when harmful generations are extremely rare.

• Applies Clopper–Pearson confidence intervals to derive probably approximately correct (PAC) harm-probability bounds.
• Prioritizes risky branches of the autoregressive generation tree using latent-space features.
• Formally guarantees that computed lower bounds do not exceed the true harmful-output probability.
• Demonstrates non-trivial lower bounds for state-of-the-art LLMs.