Structured Transforms for Low-Overhead Quantization of Language Models
Ranking
Observed public metrics from 1 member.
Merged summary
TL;DR - Kashin-DCT is a language-model weight quantization method that replaces costly dense random transforms with sign-randomized DCTs, cutting each iteration from $\mathcal{O}(N^2)$ to $\mathcal{O}(N \log N)$. It enables efficient 4-bit-per-channel compression while improving numerical stability on configurations where competing methods fail.
- A greedy alternating-update algorithm produces the four-peak distributions needed to cluster each factor into stable 2-bit codes.
- Closed-form cluster-center initialization eliminates the multi-restart k-means bottleneck used by earlier approaches.
- Combined with OPTQ-style error compensation and QuIP-style incoherence preprocessing, the JAX implementation is competitive on OPT, Llama-2, and Pythia models with favorable wall-clock scaling.
- The bounded-$\ell_\infty$ factorization remains near FP16 perplexity in reported Pythia-6.9B and Mistral-7B stress tests where QuIP variants diverge or encounter NaNs.
Sources (1)
Structured Transforms for Low-Overhead Quantization of Language Models
TL;DR - Kashin-DCT is a language-model weight quantization method that replaces costly dense random transforms with sign-randomized DCTs, cutting each iteration from $\mathcal{O}(N^2)$ to $\mathcal{O}(N \log N)$. It enables efficient 4-bit-per-channel compression while improving numerical stability on configurations where competing methods fail.
- A greedy alternating-update algorithm produces the four-peak distributions needed to cluster each factor into stable 2-bit codes.
- Closed-form cluster-center initialization eliminates the multi-restart k-means bottleneck used by earlier approaches.
- Combined with OPTQ-style error compensation and QuIP-style incoherence preprocessing, the JAX implementation is competitive on OPT, Llama-2, and Pythia models with favorable wall-clock scaling.
- The bounded-$\ell_\infty$ factorization remains near FP16 perplexity in reported Pythia-6.9B and Mistral-7B stress tests where QuIP variants diverge or encounter NaNs.