AdaFlash: Adaptive Speculative Decoding via On-Policy Distilled Diffusion Drafters

TL;DR - AdaFlash accelerates speculative LLM decoding by stabilizing diffusion-based drafting and dynamically selecting draft length. It reports up to roughly 66% higher throughput than prior state-of-the-art methods, particularly under high concurrency.

• Uses reverse-KL on-policy distillation to reduce domain-level variance and improve training stability.
• Adds an adaptive length head to address position-dependent token quality and reduce target-model verification costs.
• Retains single-pass parallel drafting while mitigating the variability introduced by bidirectional attention.
• Consistently improves deployment speedup across experiments, with the largest gains in high-concurrency settings.