Every Expert Counts: ExactMoE for Memory-Efficient W4A16 Inference

TL;DR - ExactMoE enables memory-efficient W4A16 inference for sparse MoE models by keeping quantized experts in host memory and caching routed experts on the GPU. It cuts peak GPU memory by 87.04% while preserving 81.85% of BF16 throughput and 99.23% of baseline accuracy.

• Quantizes only expert weights to group-128 INT4; all non-expert components remain BF16.
• Preserves every expert and the original top-k routing, without pruning, substitution, or CPU execution.
• On one NVIDIA L4, a 16-slot cache uses 1.836 GiB versus 14.168 GiB for BF16.
• A fully resident 64-slot setup reaches 31.923 tokens/s, exceeding BF16’s 21.662 tokens/s.