Every Expert Counts: ExactMoE for Memory-Efficient W4A16 Inference
Ranking
Overall
79
Content
95
Popularity
41
Observed public metrics from 1 member.
Merged summary
TL;DR - ExactMoE enables memory-efficient W4A16 inference for sparse MoE models by keeping quantized experts in host memory and caching routed experts on the GPU. It cuts peak GPU memory by 87.04% while preserving 81.85% of BF16 throughput and 99.23% of baseline accuracy.
- Quantizes only expert weights to group-128 INT4; all non-expert components remain BF16.
- Preserves every expert and the original top-k routing, without pruning, substitution, or CPU execution.
- On one NVIDIA L4, a 16-slot cache uses 1.836 GiB versus 14.168 GiB for BF16.
- A fully resident 64-slot setup reaches 31.923 tokens/s, exceeding BF16’s 21.662 tokens/s.
Sources (1)
Every Expert Counts: ExactMoE for Memory-Efficient W4A16 Inference
Public signals
Semantic Scholar citations 0 · Semantic Scholar influential citations 0
TL;DR - ExactMoE enables memory-efficient W4A16 inference for sparse MoE models by keeping quantized experts in host memory and caching routed experts on the GPU. It cuts peak GPU memory by 87.04% while preserving 81.85% of BF16 throughput and 99.23% of baseline accuracy.
- Quantizes only expert weights to group-128 INT4; all non-expert components remain BF16.
- Preserves every expert and the original top-k routing, without pruning, substitution, or CPU execution.
- On one NVIDIA L4, a 16-slot cache uses 1.836 GiB versus 14.168 GiB for BF16.
- A fully resident 64-slot setup reaches 31.923 tokens/s, exceeding BF16’s 21.662 tokens/s.