‹ back to 2026-07-22

Text Template Tokens Are Implicit Semantic Registers in Diffusion Transformers

arXiv cs.CV Multimodal & Generative Maohua Li, Qirui Li, Yanke Zhou, Yiduo Li, Zhaosheng Chi, Chao Xu, Cuifeng Shen, Yixuan Xu, Hanlin Tang, Kan Liu, Tao Lan, Lin Qu, Shao-Qun Zhang 2026-07-21

TL;DR - A causal analysis of text-to-image diffusion transformers finds that structural text-template tokens become implicit semantic registers that preserve object identity during denoising. The finding also enables training-free attention-head pruning that cuts 20% of attention FLOPs with a 1.4-point GenEval drop.

Template tokens begin with little prompt-specific information but become dominant image-to-text attention sinks inside the DiT.
They acquire identity indirectly: prompt semantics enter image latents first, then flow back into template tokens.
DiT heads separate semantic routing from visual synthesis, with depth progressing through identity formation, propagation, and refinement.
Heads attending most strongly to prompt tokens are largely dispensable and can be pruned using the proposed rule.

view merged work →