PolyQ: Codesigning End-to-End Quantization Framework for Scalable Edge CPU LLM Inference
Merged summary
TL;DR - PolyQ is a CPU-oriented compiler/quantization co-design that enables fine-grained, activation-aware channel-wise bit allocation for LLM inference, making fractional-bit deployment practical and efficient on edge CPUs.
- Assigns per-channel bit-widths from {2,3,4,8,16} under a user-specified average-bit budget, then a compile-time compiler permutes/clusters channels into bit-homogeneous blocks with SIMD- and LUT-compatible kernels, keeping layout regularization off the runtime path.
- On WikiText-2 across Falcon-H1-3B, Llama2-13B, and Qwen3-32B, it delivers stable quality scaling from 3–6 bits and improves perplexity by 2.4–32.1% over prior methods at a 3-bit target.
- End-to-end tests on workstation, laptop, and mobile CPUs cut activation reorder traffic by up to 70.8%, with prefill latency and decode throughput scaling nearly proportionally to the bit budget.
- Energy/token overhead stays below 2% versus an optimized LUT-based back-end, showing fractional-bit CPU deployment is predictable and energy-efficient.
Sources (1)
PolyQ: Codesigning End-to-End Quantization Framework for Scalable Edge CPU LLM Inference
TL;DR - PolyQ is a CPU-oriented compiler/quantization co-design that enables fine-grained, activation-aware channel-wise bit allocation for LLM inference, making fractional-bit deployment practical and efficient on edge CPUs.
- Assigns per-channel bit-widths from {2,3,4,8,16} under a user-specified average-bit budget, then a compile-time compiler permutes/clusters channels into bit-homogeneous blocks with SIMD- and LUT-compatible kernels, keeping layout regularization off the runtime path.
- On WikiText-2 across Falcon-H1-3B, Llama2-13B, and Qwen3-32B, it delivers stable quality scaling from 3–6 bits and improves perplexity by 2.4–32.1% over prior methods at a 3-bit target.
- End-to-end tests on workstation, laptop, and mobile CPUs cut activation reorder traffic by up to 70.8%, with prefill latency and decode throughput scaling nearly proportionally to the bit budget.
- Energy/token overhead stays below 2% versus an optimized LUT-based back-end, showing fractional-bit CPU deployment is predictable and energy-efficient.