ResKV: Reconstructing Omitted Attention Contributions for Fixed-Budget KV Cache Compression

TL;DR - ResKV is a fixed-budget KV cache compression method that splits the budget into an exact main cache plus a compact "residual" cache reconstructing the attention mass of evicted tokens, improving long-context inference quality at equal memory cost.

• Frames information lost by eviction as residual statistics in both the numerator and denominator of softmax attention, rather than as discarded tokens.
• Residual entries join the same softmax normalization as main-cache tokens, restoring both numerator and denominator mass instead of applying a post-hoc correction.
• Allocation is set per layer and per KV head via a construction-time validation proxy, with a decode-time dynamic gate adjusting residual contribution per query.
• Reported broad gains on LongBench and RULER across query-aware/agnostic settings, multiple backbones, budgets, and baselines, while preserving peak memory and long-context decode throughput.