Token Radius Attention for Efficient Video Generation
Ranking
Overall
67
Content
80
Popularity
36
Observed public metrics from 1 member.
Merged summary
TL;DR - A training-free sparse-attention method, Token Radius Attention (TRA), cuts the quadratic cost of 3D self-attention in Video Diffusion Transformers by allocating per-query compute budgets from attention entropy. It delivers ~1.6–2x inference speedup on production video models without retraining.
- Core observation: retained attention density varies per query but correlates log-linearly with attention entropy, and dominant interactions form query-centered neighborhoods with token-dependent radii — unlike head/block-level methods that share one budget across queries.
- TRA maps query entropy to an analytic token budget, then converts it into a temporally decayed radius, avoiding explicit key ranking/sorting overhead.
- Systems-level optimizations: fused entropy extraction, warm-up reuse across steps, and block-sparse mask construction to keep overhead low.
- Reported results: 9–19% of attention interactions retained and 1.56x–2.05x speedup across seven Wan2.1, Wan2.2, and HunyuanVideo T2V/I2V configurations with competitive quality; code released.
Sources (1)
Token Radius Attention for Efficient Video Generation
Public signals
Semantic Scholar citations 0 · Semantic Scholar influential citations 0
TL;DR - A training-free sparse-attention method, Token Radius Attention (TRA), cuts the quadratic cost of 3D self-attention in Video Diffusion Transformers by allocating per-query compute budgets from attention entropy. It delivers ~1.6–2x inference speedup on production video models without retraining.
- Core observation: retained attention density varies per query but correlates log-linearly with attention entropy, and dominant interactions form query-centered neighborhoods with token-dependent radii — unlike head/block-level methods that share one budget across queries.
- TRA maps query entropy to an analytic token budget, then converts it into a temporally decayed radius, avoiding explicit key ranking/sorting overhead.
- Systems-level optimizations: fused entropy extraction, warm-up reuse across steps, and block-sparse mask construction to keep overhead low.
- Reported results: 9–19% of attention interactions retained and 1.56x–2.05x speedup across seven Wan2.1, Wan2.2, and HunyuanVideo T2V/I2V configurations with competitive quality; code released.