UNIFUSION: Adapting Autoregressive Language Models into Discrete Diffusion under a Unified Reverse-Rate Objective
Merged summary
TL;DR - UNIFUSION converts pretrained autoregressive language models into uniform-noise diffusion models under a unified reverse-rate objective, allowing every token to remain editable during generation. It offers a shared interface across diffusion formulations and improves generation-quality and benchmark results over compared diffusion models.
- Unifies SEDD, MDLM/GIDD, M2S, and Neural CTMC conditional losses as a generalized KL objective over reverse rates.
- Converts clean-token predictions into multiple diffusion parameterizations, enabling switches between mask and uniform corruption kernels.
- Continually pretrains 124M- and 355M-parameter GPT-2 checkpoints for uniform-noise diffusion.
- Shows improving GenPPL–entropy trade-offs with larger sampling budgets and leads compared diffusion models on WinoGrande, SIQA, and BBH at both scales.
Sources (1)
UNIFUSION: Adapting Autoregressive Language Models into Discrete Diffusion under a Unified Reverse-Rate Objective
TL;DR - UNIFUSION converts pretrained autoregressive language models into uniform-noise diffusion models under a unified reverse-rate objective, allowing every token to remain editable during generation. It offers a shared interface across diffusion formulations and improves generation-quality and benchmark results over compared diffusion models.
- Unifies SEDD, MDLM/GIDD, M2S, and Neural CTMC conditional losses as a generalized KL objective over reverse rates.
- Converts clean-token predictions into multiple diffusion parameterizations, enabling switches between mask and uniform corruption kernels.
- Continually pretrains 124M- and 355M-parameter GPT-2 checkpoints for uniform-noise diffusion.
- Shows improving GenPPL–entropy trade-offs with larger sampling budgets and leads compared diffusion models on WinoGrande, SIQA, and BBH at both scales.