LLM Inference Under Bursty Workload Distribution: Modifying the WAIT Algorithm

TL;DR - An arXiv preprint extending the WAIT LLM-serving scheduler to handle bursty, time-varying request traffic instead of the usual constant-rate Poisson assumption. It matters because real inference workloads are dynamic, and scheduling tuned for a fixed arrival rate leaves throughput on the table.

• Argues prior LLM scheduling work assumes stationary Poisson arrivals, which misrepresents real-world bursty traffic patterns.
• Adds a lightweight extension to the state-of-the-art WAIT algorithm that performs online estimation of request intensity from observed interarrival times, requiring no prior traffic knowledge.
• Evaluated in simulation using Markov Modulated Poisson Process (MMPP) synthetic workloads with diverse request types.
• Reports higher throughput than Sarathi-Serve, ORCA, and vLLM in the evaluated low arrival-rate shift scenarios, with comparable latency; results are simulation-only, not measured on a real serving stack.