一个请求错配了GPU，AI推理可能慢100倍

TL;DR - AI inference routing must account for model placement and KV-cache locality, not merely available GPU capacity; sending a follow-up request to the wrong GPU can force costly context recomputation and increase latency by orders of magnitude.

• Identical GPUs are not interchangeable when only one holds the conversation’s KV cache and loaded model state.
• Kubernetes scheduling and routing are evolving to consider GPU type, memory, queue depth, model placement, and cache state.
• AI proxies are becoming decision and workflow layers that handle model selection, cache-aware routing, safety checks, and cluster state.
• Agent workflows also require contextual, fine-grained permissions because static access controls cannot reliably constrain dynamic tool chains.