UNIFUSION: Adapting Autoregressive Language Models into Discrete Diffusion under a Unified Reverse-Rate Objective

TL;DR - UNIFUSION converts pretrained autoregressive language models into uniform-noise diffusion models under a unified reverse-rate objective, allowing every token to remain editable during generation. It offers a shared interface across diffusion formulations and improves generation-quality and benchmark results over compared diffusion models.

• Unifies SEDD, MDLM/GIDD, M2S, and Neural CTMC conditional losses as a generalized KL objective over reverse rates.
• Converts clean-token predictions into multiple diffusion parameterizations, enabling switches between mask and uniform corruption kernels.
• Continually pretrains 124M- and 355M-parameter GPT-2 checkpoints for uniform-noise diffusion.
• Shows improving GenPPL–entropy trade-offs with larger sampling budgets and leads compared diffusion models on WinoGrande, SIQA, and BBH at both scales.