Depth-adaptive Inference of Looped Language Models via Continuous Depth Batching
Ranking
Overall
69
Content
80
Popularity
43
Observed public metrics from 1 member.
Merged summary
TL;DR - An arXiv paper introducing continuous depth batching (CDB), an inference scheduling scheme that makes depth-adaptive looped language models practical to serve efficiently. It matters because looped LMs promise per-token compute allocation, but that adaptivity breaks the uniform forward pass assumed by frameworks like vLLM.
- Problem: token-level schedulers (e.g. vLLM) can't remove tokens mid-forward-pass, so variable loop counts per token block standard batching; loop-level scheduling was proposed but never implemented end to end.
- Core difficulty addressed: looped architectures include non-looped boundary stages (token embedding, LM head) that must be scheduled at a different frequency than the loop body.
- CDB design: schedules at individual loop-iteration granularity, uses separate priority queues for boundary stages vs. loop steps, makes exit decisions one step ahead, and overlaps scheduling work with GPU compute.
- Results on Ouro 1.4B and Huginn 3.5B: up to 99% of the theoretical max adaptive-depth speed-up, 1.5–1.9× higher offline throughput, and 45–90% lower normalized latency under dynamic serving load.
Sources (1)
Depth-adaptive Inference of Looped Language Models via Continuous Depth Batching
Public signals
Semantic Scholar citations 0 · Semantic Scholar influential citations 0
TL;DR - An arXiv paper introducing continuous depth batching (CDB), an inference scheduling scheme that makes depth-adaptive looped language models practical to serve efficiently. It matters because looped LMs promise per-token compute allocation, but that adaptivity breaks the uniform forward pass assumed by frameworks like vLLM.
- Problem: token-level schedulers (e.g. vLLM) can't remove tokens mid-forward-pass, so variable loop counts per token block standard batching; loop-level scheduling was proposed but never implemented end to end.
- Core difficulty addressed: looped architectures include non-looped boundary stages (token embedding, LM head) that must be scheduled at a different frequency than the loop body.
- CDB design: schedules at individual loop-iteration granularity, uses separate priority queues for boundary stages vs. loop steps, makes exit decisions one step ahead, and overlaps scheduling work with GPU compute.
- Results on Ouro 1.4B and Huginn 3.5B: up to 99% of the theoretical max adaptive-depth speed-up, 1.5–1.9× higher offline throughput, and 45–90% lower normalized latency under dynamic serving load.