🛰️ Daily AI Frontier
‹ back to 2026-08-10

Reducing Pretraining-Generation Mismatch in Diffusion Language Models

Research LLMs & Foundation Models

Ranking

Overall 69
Content 80
Popularity 43

Observed public metrics from 1 member.

Representative image for Reducing Pretraining-Generation Mismatch in Diffusion Language Models

Merged summary

TL;DR - An arXiv preprint identifying a pretraining-generation mismatch in diffusion language models, where native dLLM pretraining corrupts prompt and continuation tokens together, and proposing PCD (Prefix-Conditioned Diffusion) to fix it. It matters because it recovers part of the dLLM continuation gap purely via a training-objective change, with no inference-time modifications.

  • PCD combines autoregressive supervision on the clean prefix with no-shift denoising on the suffix, implemented by altering the attention mask, corruption mask, and label construction during continued pretraining — no AR decoder, verifier, or new inference mode required.
  • The design makes the local training interface resemble how block-diffusion models are actually queried at evaluation time, restoring the clean-prefix interface needed for prompt-conditioned generation.
  • The authors disentangle intra-sample prefix conditioning from inter-sample objective mixing, isolating the local alignment signal from the optional batch-level mixing knob.
  • Reported gains over same-family native dLLM stable baselines: +2.56 points (4.2% relative) on the LLaDA2-Mini six-benchmark average, and +4.86 points (14.2% relative) in the primary Qwen-1.7B mechanism comparison.

Sources (1)

Reducing Pretraining-Generation Mismatch in Diffusion Language Models

arXiv cs.CL Xiaocheng Lu, Huabin Liu, Song Guo, Jianguo Li 2026-08-10 arXiv:2608.09424
Public signals Semantic Scholar citations 0 · Semantic Scholar influential citations 0
Providers: Hugging Face · N/A OpenAlex · N/A Publisher · N/A Semantic Scholar · Citations 0 · Influential citations 0 X · N/A Fetched 2026-09-02 14:23:55.446492 UTC

TL;DR - An arXiv preprint identifying a pretraining-generation mismatch in diffusion language models, where native dLLM pretraining corrupts prompt and continuation tokens together, and proposing PCD (Prefix-Conditioned Diffusion) to fix it. It matters because it recovers part of the dLLM continuation gap purely via a training-objective change, with no inference-time modifications.

  • PCD combines autoregressive supervision on the clean prefix with no-shift denoising on the suffix, implemented by altering the attention mask, corruption mask, and label construction during continued pretraining — no AR decoder, verifier, or new inference mode required.
  • The design makes the local training interface resemble how block-diffusion models are actually queried at evaluation time, restoring the clean-prefix interface needed for prompt-conditioned generation.
  • The authors disentangle intra-sample prefix conditioning from inter-sample objective mixing, isolating the local alignment signal from the optional batch-level mixing knob.
  • Reported gains over same-family native dLLM stable baselines: +2.56 points (4.2% relative) on the LLaDA2-Mini six-benchmark average, and +4.86 points (14.2% relative) in the primary Qwen-1.7B mechanism comparison.
item →