LLM Inference Under Bursty Workload Distribution: Modifying the WAIT Algorithm
Ranking
Overall
58
Content
65
Popularity
42
Observed public metrics from 1 member.
Merged summary
TL;DR - An arXiv preprint extending the WAIT LLM-serving scheduler to handle bursty, time-varying request traffic instead of the usual constant-rate Poisson assumption. It matters because real inference workloads are dynamic, and scheduling tuned for a fixed arrival rate leaves throughput on the table.
- Argues prior LLM scheduling work assumes stationary Poisson arrivals, which misrepresents real-world bursty traffic patterns.
- Adds a lightweight extension to the state-of-the-art WAIT algorithm that performs online estimation of request intensity from observed interarrival times, requiring no prior traffic knowledge.
- Evaluated in simulation using Markov Modulated Poisson Process (MMPP) synthetic workloads with diverse request types.
- Reports higher throughput than Sarathi-Serve, ORCA, and vLLM in the evaluated low arrival-rate shift scenarios, with comparable latency; results are simulation-only, not measured on a real serving stack.
Sources (1)
LLM Inference Under Bursty Workload Distribution: Modifying the WAIT Algorithm
Public signals
Semantic Scholar citations 0 · Semantic Scholar influential citations 0
TL;DR - An arXiv preprint extending the WAIT LLM-serving scheduler to handle bursty, time-varying request traffic instead of the usual constant-rate Poisson assumption. It matters because real inference workloads are dynamic, and scheduling tuned for a fixed arrival rate leaves throughput on the table.
- Argues prior LLM scheduling work assumes stationary Poisson arrivals, which misrepresents real-world bursty traffic patterns.
- Adds a lightweight extension to the state-of-the-art WAIT algorithm that performs online estimation of request intensity from observed interarrival times, requiring no prior traffic knowledge.
- Evaluated in simulation using Markov Modulated Poisson Process (MMPP) synthetic workloads with diverse request types.
- Reports higher throughput than Sarathi-Serve, ORCA, and vLLM in the evaluated low arrival-rate shift scenarios, with comparable latency; results are simulation-only, not measured on a real serving stack.