ECCV Spotlight｜王利民团队提出UniDDT ：让多模态理解和生成不再互相拖累

TL;DR - UniDDT (ECCV Spotlight, Wang Limin's NJU MCG team) is a unified multimodal model that shares one semantic encoding path for understanding and image generation while offloading high-frequency visual detail to a separate diffusion decoder, showing the two tasks need not trade off against each other.

• Architecture is Noisy ViT + LLM + Diffusion Decoder: the ViT ingests noisy images/latents plus timestep (warm-started by representation distillation from a pretrained teacher), the LLM fuses prompt with current visual state into refined visual features, and a lightweight diffusion decoder predicts velocity — extending DDT's decoupling of condition encoder and velocity decoder.
• Core premise: semantic encoding during denoising is essentially "understanding a noisy image," so understanding is folded into every diffusion timestep rather than bolted on before/after generation.
• Three-stage training (Noisy ViT warmup → decoder warmup with ViT/LLM frozen → joint training → post-training); post-training uses the understanding branch to score semantic consistency of intermediate noisy states instead of relying on an external reward model.
• Reported results for VLM-UniDDT: GenEval 0.87, DPGBench 86.9, MME perception 1699.5, SEEDBench 76.5. Latent (VAE) space was chosen over pixel space — pixel matched on understanding but lagged badly in pretraining generation due to lacking GAN/LPIPS visual priors, though the authors argue pixel has large post-training potential.