D-cut: Adaptive Verification Depth Pruning for Batched Speculative Decoding

TL;DR - D-Cut is an adaptive pruning method for batched speculative decoding that reallocates the verification budget across concurrent requests, restoring and improving LLM inference speedups under high concurrency where long-draft methods otherwise stall.

• Performs cross-request pruning: since acceptance lengths vary across concurrent requests, it allocates verification budget adaptively based on per-request draft confidence.
• Uses a runtime cost model to tune pruning depth to the deployment environment (GPU architecture, parallelism strategy), since verification cost is environment-dependent.
• Under high concurrency, raises average speedup from 1.26× to 1.65×, and restores acceleration in dense-model setups where long-draft baselines are slower than autoregressive decoding.
• Achieves up to 3.0× speedup over autoregressive decoding on mixture-of-experts (MoE) models; evaluated on both dense and MoE models.