Language models suffer from a curse of ambiguity

TL;DR - This paper identifies a “curse of ambiguity”: neural networks learn uncertain next-token probability distributions less accurately than concentrated ones. This matters as LLM training increasingly depends on sampling from learned distributions.

• Ambiguous distributions require greater model capacity and larger embeddings to store and represent accurately.
• They take more optimization steps to fit and amplify noise during token sampling.
• Controlled synthetic experiments support the theory, while models trained on real data show similar patterns.
• The analysis offers a framework for judging when an LLM’s output distribution is trustworthy.