WIDE: Boosting Adaptive LLM Inference via Token-level Dynamic Width Pruning

TL;DR - WIDE dynamically prunes attention-head and FFN-channel groups for each token, improving LLM inference efficiency while retaining more quality than coarse-grained pruning. Its kernel co-design delivers practical prefill and decoding acceleration.

• Supports token-level dynamic width pruning during both prefill and decoding.
• Uses a two-stage differentiable training pipeline to learn token-wise sparse execution.
• At 50% sparsity, reports a 55.1% performance boost over dynamic depth pruning under calibration-only settings.
• Achieves up to 1.98× prefill and 4.95× decoding kernel speedups, with 1.68× and 1.55× end-to-end acceleration.