AURORA-LM: Autoencoding Unified Representation for Continuous-Latent Diffusion Language Modeling

TL;DR - AURORA-LM is a continuous-latent diffusion language model that keeps a high-capacity, decodable text latent instead of compressing it, and adapts the diffusion model to that harder distribution. It matters because it pushes non-autoregressive, continuous-space text generation closer to practical quality without sacrificing token-level fidelity.

• Two-part design: a Query-based Encoder-Decoder builds a prefix-aligned, high-capacity latent sequence, and a Block-causal Diffusion Transformer learns its distribution via flow matching — generating blocks left-to-right while denoising positions within a block in parallel.
• To handle the harder latent, it restricts only the noisy-input pathway while keeping the full clean-latent prediction target, so decoder-facing capacity stays intact at full latent width.
• Adds noise-level distribution calibration tied to latent width, plus self-trajectory consistency to reconcile independently sampled training noise with iterative inference-time denoising.
• Reports best results among evaluated continuous/diffusion LMs on OpenWebText free generation and XSum summarization; scaling to 1B params (~1500 EFLOPs) beats a larger public latent-diffusion LM under a matched protocol, all trained on Ascend NPUs.