Oilbird: Training-Free Speculative Decoding with Keys the Verifier Already Computes

TL;DR - Oilbird is a training-free speculative decoding method that uses verifier-computed hidden states to retrieve semantically matching drafts missed by exact suffix lookup. It improves accepted draft length and reaches 4.4× autoregressive decoding speed on API-Bank.

• Re-keys an existing context pool with hidden states already computed at each committed token.
• Targets tool-calling workloads where repeated requests differ mainly in newly generated values.
• Improves accepted length by 24–29% across three published drafters at matched pool size and budget.
• Achieves 4.4× speedup on API-Bank, versus 3.9× for the strongest training-free baseline and 2.0× for EAGLE-3.