ECCV Spotlight｜王利民团队提出UniDDT ：让多模态理解和生成不再互相拖累
Ranking
Overall
71
Content
80
Popularity
N/A
No observed public metrics; popularity remains neutral/archived.
Merged summary
TL;DR - UniDDT (ECCV Spotlight, Wang Limin's NJU MCG team) is a unified multimodal model that shares one semantic encoding path for understanding and image generation while offloading high-frequency visual detail to a separate diffusion decoder, showing the two tasks need not trade off against each other.
- Architecture is Noisy ViT + LLM + Diffusion Decoder: the ViT ingests noisy images/latents plus timestep (warm-started by representation distillation from a pretrained teacher), the LLM fuses prompt with current visual state into refined visual features, and a lightweight diffusion decoder predicts velocity — extending DDT's decoupling of condition encoder and velocity decoder.
- Core premise: semantic encoding during denoising is essentially "understanding a noisy image," so understanding is folded into every diffusion timestep rather than bolted on before/after generation.
- Three-stage training (Noisy ViT warmup → decoder warmup with ViT/LLM frozen → joint training → post-training); post-training uses the understanding branch to score semantic consistency of intermediate noisy states instead of relying on an external reward model.
- Reported results for VLM-UniDDT: GenEval 0.87, DPGBench 86.9, MME perception 1699.5, SEEDBench 76.5. Latent (VAE) space was chosen over pixel space — pixel matched on understanding but lagged badly in pretraining generation due to lacking GAN/LPIPS visual priors, though the authors argue pixel has large post-training potential.
Sources (1)
ECCV Spotlight｜王利民团队提出UniDDT ：让多模态理解和生成不再互相拖累
Public signals
N/A
TL;DR - UniDDT (ECCV Spotlight, Wang Limin's NJU MCG team) is a unified multimodal model that shares one semantic encoding path for understanding and image generation while offloading high-frequency visual detail to a separate diffusion decoder, showing the two tasks need not trade off against each other.
- Architecture is Noisy ViT + LLM + Diffusion Decoder: the ViT ingests noisy images/latents plus timestep (warm-started by representation distillation from a pretrained teacher), the LLM fuses prompt with current visual state into refined visual features, and a lightweight diffusion decoder predicts velocity — extending DDT's decoupling of condition encoder and velocity decoder.
- Core premise: semantic encoding during denoising is essentially "understanding a noisy image," so understanding is folded into every diffusion timestep rather than bolted on before/after generation.
- Three-stage training (Noisy ViT warmup → decoder warmup with ViT/LLM frozen → joint training → post-training); post-training uses the understanding branch to score semantic consistency of intermediate noisy states instead of relying on an external reward model.
- Reported results for VLM-UniDDT: GenEval 0.87, DPGBench 86.9, MME perception 1699.5, SEEDBench 76.5. Latent (VAE) space was chosen over pixel space — pixel matched on understanding but lagged badly in pretraining generation due to lacking GAN/LPIPS visual priors, though the authors argue pixel has large post-training potential.