C$^2$KV: Compressed and Composable KV Cache Reuse for Efficient LLM Inference

TL;DR - C²KV compresses and composes reusable, non-prefix KV caches to reduce long-context LLM inference costs. It reports up to 17× inference speedup while preserving generation quality.

• Learns position-agnostic KV representations that can be reused and concatenated across contexts.
• Uses a lightweight sidecar Extractor with learnable compression tokens and structured attention.
• Keeps the base model frozen, avoiding architectural modifications.
• Jointly trains compression and concatenation to prevent the accuracy degradation seen with naive combinations.