Messages, Not Tokens: Grounded Coresets for Faithful VLM Compression

TL;DR - An arXiv cs.CV preprint reframes visual token pruning in vision-language models as building a coreset for the collective attention messages the decoder receives, rather than scoring tokens independently. It matters because it lets VLMs drop ~80% of visual tokens while retaining near-full accuracy, cutting KV-cache and inference cost.

• Critiques score-based Top-K pruning: independent importance scores over-cover one salient region, miss sparse complementary evidence, and throw away the signed message contributed by removed tokens.
• Proposes Grounded Message Coreset Pruning (GMC), training-free, allocating support jointly across query-grounded, appearance, and coordinate-aware evidence, then transporting discarded states onto selected representatives at their original multimodal positions before physical compaction and native attention resume.
• Decomposes the problem into carrier selection (covering message modes) and population realization on those carriers, with derived error bounds tied to signed-message distortion, visual innovation, and candidate-margin stability.
• Reported results: GMC-H2 keeps 97.78% of full-model mean capability on Qwen2.5-VL-7B with 80.2% fewer visual tokens; GMC-L16 reaches 100.36%. Ablations attribute gains to both collective support and population realization.