Reducing Pretraining-Generation Mismatch in Diffusion Language Models

TL;DR - An arXiv preprint identifying a pretraining-generation mismatch in diffusion language models, where native dLLM pretraining corrupts prompt and continuation tokens together, and proposing PCD (Prefix-Conditioned Diffusion) to fix it. It matters because it recovers part of the dLLM continuation gap purely via a training-objective change, with no inference-time modifications.

• PCD combines autoregressive supervision on the clean prefix with no-shift denoising on the suffix, implemented by altering the attention mask, corruption mask, and label construction during continued pretraining — no AR decoder, verifier, or new inference mode required.
• The design makes the local training interface resemble how block-diffusion models are actually queried at evaluation time, restoring the clean-prefix interface needed for prompt-conditioned generation.
• The authors disentangle intra-sample prefix conditioning from inter-sample objective mixing, isolating the local alignment signal from the optional batch-level mixing knob.
• Reported gains over same-family native dLLM stable baselines: +2.56 points (4.2% relative) on the LLaDA2-Mini six-benchmark average, and +4.86 points (14.2% relative) in the primary Qwen-1.7B mechanism comparison.