Ready Cohorts: Bounding GPU Opportunity and Avoiding Host Round Trips in LLM-Agent Control

TL;DR - This paper identifies when LLM-agent control transitions can benefit from GPU batching and device-resident routing. It shows that dynamic cohort packing captures substantially more GPU opportunity while avoiding host round trips consistently improves latency.

• At 100,000 active sessions, dynamic packing achieved a 43.00% offline share versus 30.19% for fixed partitions, recovering 81.83% of lost opportunity.
• Keeping GPU-computed routing decisions on-device was faster across all 36 configurations, with median speedups of 1.19×–2.39×.
• All 14,557,440 tested batched invocations matched an independently implemented host oracle.
• Device graphs that retained host decision-making were slower in all 60 tested configurations.