ExaGEMM: Exploration Framework for CPU-Driven ML Inference via Associative In-Register Computing for Low-Bit GEMM
Merged summary
TL;DR - ExaGEMM is a codesign and exploration framework for running low-bit GEMM efficiently on CPUs via register-resident lookup-table (LUT) execution, addressing the poor fit between very-low-bit ML inference and conventional CPU datapaths. It matters because low-bit and mixed-precision GEMM is increasingly central to efficient LLM inference, yet CPU support has been ad hoc.
- Targets fragmented low-bit regimes (1/2/4-bit weights with varying activation precision) under fixed SIMD and register-file budgets, framing lightweight CPU support selection as a first-class design problem.
- Key insight: existing SIMD datapaths already handle table generation and accumulation, so the only new hardware needed is an in-register select/feed mechanism with explicitly modeled cost.
- Uses analytical models (register feasibility, compute cost, memory traffic, hardware overhead) to prune the candidate space by 99.2% before simulation, then emits ISA specs, gem5 patches, and GEMM kernels for validation.
- Reports 13.29x latency improvement over software-only baselines, with workload-aware frontier selection especially beneficial for mixed-precision LLM workloads.
Sources (1)
ExaGEMM: Exploration Framework for CPU-Driven ML Inference via Associative In-Register Computing for Low-Bit GEMM
TL;DR - ExaGEMM is a codesign and exploration framework for running low-bit GEMM efficiently on CPUs via register-resident lookup-table (LUT) execution, addressing the poor fit between very-low-bit ML inference and conventional CPU datapaths. It matters because low-bit and mixed-precision GEMM is increasingly central to efficient LLM inference, yet CPU support has been ad hoc.
- Targets fragmented low-bit regimes (1/2/4-bit weights with varying activation precision) under fixed SIMD and register-file budgets, framing lightweight CPU support selection as a first-class design problem.
- Key insight: existing SIMD datapaths already handle table generation and accumulation, so the only new hardware needed is an in-register select/feed mechanism with explicitly modeled cost.
- Uses analytical models (register feasibility, compute cost, memory traffic, hardware overhead) to prune the candidate space by 99.2% before simulation, then emits ISA specs, gem5 patches, and GEMM kernels for validation.
- Reports 13.29x latency improvement over software-only baselines, with workload-aware frontier selection especially beneficial for mixed-precision LLM workloads.