Text Template Tokens Are Implicit Semantic Registers in Diffusion Transformers
Merged summary
TL;DR - A causal analysis of text-to-image diffusion transformers finds that structural text-template tokens become implicit semantic registers that preserve object identity during denoising. The finding also enables training-free attention-head pruning that cuts 20% of attention FLOPs with a 1.4-point GenEval drop.
- Template tokens begin with little prompt-specific information but become dominant image-to-text attention sinks inside the DiT.
- They acquire identity indirectly: prompt semantics enter image latents first, then flow back into template tokens.
- DiT heads separate semantic routing from visual synthesis, with depth progressing through identity formation, propagation, and refinement.
- Heads attending most strongly to prompt tokens are largely dispensable and can be pruned using the proposed rule.
Sources (1)
Text Template Tokens Are Implicit Semantic Registers in Diffusion Transformers
TL;DR - A causal analysis of text-to-image diffusion transformers finds that structural text-template tokens become implicit semantic registers that preserve object identity during denoising. The finding also enables training-free attention-head pruning that cuts 20% of attention FLOPs with a 1.4-point GenEval drop.
- Template tokens begin with little prompt-specific information but become dominant image-to-text attention sinks inside the DiT.
- They acquire identity indirectly: prompt semantics enter image latents first, then flow back into template tokens.
- DiT heads separate semantic routing from visual synthesis, with depth progressing through identity formation, propagation, and refinement.
- Heads attending most strongly to prompt tokens are largely dispensable and can be pruned using the proposed rule.