D-cut: Adaptive Verification Depth Pruning for Batched Speculative Decoding
Merged summary
TL;DR - D-Cut is an adaptive pruning method for batched speculative decoding that reallocates the verification budget across concurrent requests, restoring and improving LLM inference speedups under high concurrency where long-draft methods otherwise stall.
- Performs cross-request pruning: since acceptance lengths vary across concurrent requests, it allocates verification budget adaptively based on per-request draft confidence.
- Uses a runtime cost model to tune pruning depth to the deployment environment (GPU architecture, parallelism strategy), since verification cost is environment-dependent.
- Under high concurrency, raises average speedup from 1.26× to 1.65×, and restores acceleration in dense-model setups where long-draft baselines are slower than autoregressive decoding.
- Achieves up to 3.0× speedup over autoregressive decoding on mixture-of-experts (MoE) models; evaluated on both dense and MoE models.
Sources (1)
D-cut: Adaptive Verification Depth Pruning for Batched Speculative Decoding
TL;DR - D-Cut is an adaptive pruning method for batched speculative decoding that reallocates the verification budget across concurrent requests, restoring and improving LLM inference speedups under high concurrency where long-draft methods otherwise stall.
- Performs cross-request pruning: since acceptance lengths vary across concurrent requests, it allocates verification budget adaptively based on per-request draft confidence.
- Uses a runtime cost model to tune pruning depth to the deployment environment (GPU architecture, parallelism strategy), since verification cost is environment-dependent.
- Under high concurrency, raises average speedup from 1.26× to 1.65×, and restores acceleration in dense-model setups where long-draft baselines are slower than autoregressive decoding.
- Achieves up to 3.0× speedup over autoregressive decoding on mixture-of-experts (MoE) models; evaluated on both dense and MoE models.