PagedWeight: Efficient MoE LLM Serving with Dynamic Quality-Aware Weight Quantization

TL;DR - PagedWeight is a runtime memory-management method for serving Mixture-of-Experts (MoE) LLMs that dynamically quantizes expert weights to balance weight precision against KV-cache size. It matters because it eases the GPU-memory tension in KV-cache-intensive MoE serving, improving the quality–memory–throughput tradeoff.

• Targets the tension between MoE model-weight memory and growing KV cache in memory-sensitive serving scenarios.
• Dynamically, quality-aware quantizes expert weights at runtime, navigating the accuracy/memory/throughput tradeoff.
• Reports FP16-equivalent accuracy with up to 72.0% GPU memory savings and 1.94× throughput improvement.
• Improves quality over quantization baselines by up to 39.3% at similar memory budgets, with ≤4.1% throughput loss.