OmniKVQuant: KV Cache Quantization for Omni-LLMs
Ranking
Overall
81
Content
100
Popularity
37
Observed public metrics from 1 member.
Merged summary
TL;DR - OmniKVQuant is a training-free KV-cache quantization framework for multimodal LLMs that accounts for differences across audio, video, and text representations. It enables 2-bit caches while substantially preserving performance, reducing memory demands without reconstructing a dense FP16 cache during decoding.
- Identifies temporal key drift and heterogeneous value geometry as key limitations of applying text-oriented TurboQuant to multimodal caches.
- Sets key quantization ranges over short input-stream windows and applies separate value rotations for each modality.
- Demonstrates 2-bit KV caches on Qwen2.5-Omni and Qwen3-Omni across seven audio-visual benchmarks.
- Provides a fused Triton decoding kernel that unpacks quantized values directly during attention.
Sources (1)
OmniKVQuant: KV Cache Quantization for Omni-LLMs
Public signals
Semantic Scholar citations 0 · Semantic Scholar influential citations 0
TL;DR - OmniKVQuant is a training-free KV-cache quantization framework for multimodal LLMs that accounts for differences across audio, video, and text representations. It enables 2-bit caches while substantially preserving performance, reducing memory demands without reconstructing a dense FP16 cache during decoding.
- Identifies temporal key drift and heterogeneous value geometry as key limitations of applying text-oriented TurboQuant to multimodal caches.
- Sets key quantization ranges over short input-stream windows and applies separate value rotations for each modality.
- Demonstrates 2-bit KV caches on Qwen2.5-Omni and Qwen3-Omni across seven audio-visual benchmarks.
- Provides a fused Triton decoding kernel that unpacks quantized values directly during attention.