Partition, Prompt, Aggregate: Statistical Self-Consistency in Language Models

TL;DR - This paper tests whether LLM in-context estimates obey the law of total probability, finding widespread violations of statistical self-consistency—a reference-free way to evaluate frontier models.

• Uses binary trees to recursively partition populations into finer subpopulations, prompts LLMs with verbalized subpopulation descriptions, then aggregates estimates back to population level and compares across partition granularities.
• Across multiple domains and state-of-the-art models, LLM estimates broadly violate basic probabilistic identities (prior-weighted conditionals should aggregate into marginals).
• Identifies a "macro fallacy": estimates reconstructed from fine-grained subpopulation responses often align better with human reference data than direct population-level estimates—models hold subpopulation knowledge but fail to propagate it into aggregates.
• The effect persists across tree structures and tasks and is partially recoverable via implicit prompting, positioning self-consistency as an unsaturated, reference-free evaluation criterion.