Context-weighted Discrete Flow Matching

TL;DR - This paper makes discrete flow matching context-aware, prioritizing tokens based on nearby contextual information. The method improves generation with negligible overhead and reduces OpenWebText generative perplexity by up to 63%.

• Modifies the underlying continuous-time Markov chain to incorporate local context density.
• Reweights token-level training signals with a scaled cross-entropy loss.
• Matches a strong semi-autoregressive block diffusion baseline while preserving arbitrary-order generation.
• Improves both sampling quality and training efficiency.