TPAMI | 北大 & 清华 & 复旦 提出 SparseVLM+：修正注意力偏置，让「文本引导的视觉稀疏化」更精准

TL;DR - SparseVLM+ (Peking U., Tsinghua, Fudan; TPAMI journal extension of ICML 2025's SparseVLM) is a training-free, plug-and-play method for pruning visual tokens in vision-language models that first corrects the attention signal used to rank tokens. It matters because attention-based pruning is the dominant efficiency trick for VLMs, and this work shows the attention scores themselves are systematically biased.

• Two diagnosed biases: Attention Gravity — RoPE-style position encoding makes text tokens over-attend to physically nearby visual tokens rather than semantically relevant ones (strongest in shallow layers); and Attention Sink — semantically irrelevant visual tokens absorb disproportionate attention regardless of the query.
• Fixes: gravity correction estimates the pure positional prior by running uniformly-initialized queries/keys, then divides it out of the real attention map; Priority Heads Selection keeps only the Top-K attention heads with the highest weighted cross-modal scores, discarding noisy heads.
• Video extension: Temporal Balanced Sparsification combines corrected text-relevance with a temporal diversity score (nearest-neighbor Euclidean distance to already-selected tokens), keeping Top-K tokens per frame.
• Reported results: LLaVA-1.5-7B retains 99.6% performance at 66.7% token compression (192 tokens), +1.5% over SparseVLM and +1.6% over VisPruner; 96.9% retained at 80.2% pruning; Qwen2.5-VL gains 0.5 on MathVista at ~60.5% reduction; LLaVA-OneVision-7B reaches 96.2% with only 15% of tokens vs. 94.6% for FrameFusion.