Limits of Confidence in Diffusion

TL;DR - This paper identifies a fundamental limitation in discrete diffusion samplers that generate multiple token positions independently per step: they reproduce the training distribution only when those positions are conditionally independent. This matters because standard per-sample metrics can look perfect while concealing substantial distributional error.

• No product of per-position distributions can represent a jointly dependent group of tokens.
• Per-position marginals cannot reveal group dependence, since different joint distributions may share identical marginals.
• On the synthetic ScanAndAdd task, every multi-position group selected by confidence ranking is dependent.
• The generated distribution’s total variation is (29\times) the sampling-noise floor despite per-sample metrics scoring (1.0).