FVAttn: Adaptive Sparse Attention with Runtime Load Balancing for Video Generation
Merged summary
TL;DR - FVAttn is a training-free sparse-attention system for distributed video diffusion transformers that dynamically balances uneven GPU workloads. It achieves up to 4.41× faster attention and 2.02–2.11× faster DiT inference while maintaining competitive video quality.
- Combines adaptive Top-p routing with a Top-k safety floor and video-aware block organization.
- Migrates heavy attention heads between GPUs at runtime to reduce rank-level stragglers.
- Uses idle GPU capacity to process additional high-value blocks while overlapping scheduling and communication with computation.
- Reduces average load imbalance from 1.34 to 1.08 on step-distilled Wan2.2 image-to-video generation.
Sources (1)
FVAttn: Adaptive Sparse Attention with Runtime Load Balancing for Video Generation
TL;DR - FVAttn is a training-free sparse-attention system for distributed video diffusion transformers that dynamically balances uneven GPU workloads. It achieves up to 4.41× faster attention and 2.02–2.11× faster DiT inference while maintaining competitive video quality.
- Combines adaptive Top-p routing with a Top-k safety floor and video-aware block organization.
- Migrates heavy attention heads between GPUs at runtime to reduce rank-level stragglers.
- Uses idle GPU capacity to process additional high-value blocks while overlapping scheduling and communication with computation.
- Reduces average load imbalance from 1.34 to 1.08 on step-distilled Wan2.2 image-to-video generation.