Cross-Entropy Risk Estimation for Language Models: Inconsistency Must Be Dense, and the Holdout Method Is No Exception
Ranking
Overall
75
Content
90
Popularity
41
Observed public metrics from 1 member.
Merged summary
TL;DR - This paper proves that language models’ held-out per-token cross-entropy risk cannot be consistently estimated across all possible data distributions and trained models. It identifies bounded probability flooring and thresholded risk reporting as two qualified remedies.
- Finite- and infinite-risk states can lie arbitrarily close, preventing any estimator—not only holdout averages—from being universally consistent.
- The impossibility persists with bounded expected sequence length and full-support models; under full support, inconsistent states are dense.
- With bounded context, flooring next-token probabilities makes finite risk equivalent to finite expected sequence length.
- Reporting risk only below a predetermined threshold restores consistency while retaining what model selection requires, but changes the estimation objective.
Sources (1)
Cross-Entropy Risk Estimation for Language Models: Inconsistency Must Be Dense, and the Holdout Method Is No Exception
Public signals
Semantic Scholar citations 0 · Semantic Scholar influential citations 0
TL;DR - This paper proves that language models’ held-out per-token cross-entropy risk cannot be consistently estimated across all possible data distributions and trained models. It identifies bounded probability flooring and thresholded risk reporting as two qualified remedies.
- Finite- and infinite-risk states can lie arbitrarily close, preventing any estimator—not only holdout averages—from being universally consistent.
- The impossibility persists with bounded expected sequence length and full-support models; under full support, inconsistent states are dense.
- With bounded context, flooring next-token probabilities makes finite risk equivalent to finite expected sequence length.
- Reporting risk only below a predetermined threshold restores consistency while retaining what model selection requires, but changes the estimation objective.