Depth-adaptive Inference of Looped Language Models via Continuous Depth Batching

TL;DR - An arXiv paper introducing continuous depth batching (CDB), an inference scheduling scheme that makes depth-adaptive looped language models practical to serve efficiently. It matters because looped LMs promise per-token compute allocation, but that adaptivity breaks the uniform forward pass assumed by frameworks like vLLM.

• Problem: token-level schedulers (e.g. vLLM) can't remove tokens mid-forward-pass, so variable loop counts per token block standard batching; loop-level scheduling was proposed but never implemented end to end.
• Core difficulty addressed: looped architectures include non-looped boundary stages (token embedding, LM head) that must be scheduled at a different frequency than the loop body.
• CDB design: schedules at individual loop-iteration granularity, uses separate priority queues for boundary stages vs. loop steps, makes exit decisions one step ahead, and overlaps scheduling work with GPU compute.
• Results on Ouro 1.4B and Huginn 3.5B: up to 99% of the theoretical max adaptive-depth speed-up, 1.5–1.9× higher offline throughput, and 45–90% lower normalized latency under dynamic serving load.