OmniKVQuant: KV Cache Quantization for Omni-LLMs

TL;DR - OmniKVQuant is a training-free KV-cache quantization framework for multimodal LLMs that accounts for differences across audio, video, and text representations. It enables 2-bit caches while substantially preserving performance, reducing memory demands without reconstructing a dense FP16 cache during decoding.

• Identifies temporal key drift and heterogeneous value geometry as key limitations of applying text-oriented TurboQuant to multimodal caches.
• Sets key quantization ranges over short input-stream windows and applies separate value rotations for each modality.
• Demonstrates 2-bit KV caches on Qwen2.5-Omni and Qwen3-Omni across seven audio-visual benchmarks.
• Provides a fused Triton decoding kernel that unpacks quantized values directly during attention.