HiKV: Hierarchical Importance-Aware KV Cache with Hardware Acceleration for LLM Decoding

TL;DR - HiKV is an algorithm-hardware co-design that compresses LLM KV caches using hierarchical importance filtering. It accelerates attention by up to 7.95× and reduces energy by 90% with negligible accuracy loss.

• First evicts unimportant tokens, then loads only significant elements from retained tokens.
• Uses a reconfigurable importance sorter to support both compression stages in one accelerator.
• Reduces external memory accesses by an additional 1.82–4.87× versus importance-based alternatives at equal accuracy.
• Adds 8% system area while keeping accuracy loss within 1%.