FVAttn: Adaptive Sparse Attention with Runtime Load Balancing for Video Generation

TL;DR - FVAttn is a training-free sparse-attention system for distributed video diffusion transformers that dynamically balances uneven GPU workloads. It achieves up to 4.41× faster attention and 2.02–2.11× faster DiT inference while maintaining competitive video quality.

• Combines adaptive Top-p routing with a Top-k safety floor and video-aware block organization.
• Migrates heavy attention heads between GPUs at runtime to reduce rank-level stragglers.
• Uses idle GPU capacity to process additional high-value blocks while overlapping scheduling and communication with computation.
• Reduces average load imbalance from 1.34 to 1.08 on step-distilled Wan2.2 image-to-video generation.